OK, I'll treat this as a bug for now. It does seem like it's getting
better. I wrote a quick script to list those files with zlib compression.
There are a lot of N0S and EET products but these aren't also bzip2
compressed so my code handles it. There were 46 N0B products in the last
hour that were zlib compressed:
/mnt/data/noaaport/nids/BGM/20220307_1904_N0B.nid 368774 zlib
/mnt/data/noaaport/nids/BGM/20220307_1910_N0B.nid 366048 zlib
/mnt/data/noaaport/nids/BGM/20220307_1916_N0B.nid 365302 zlib
/mnt/data/noaaport/nids/BGM/20220307_1922_N0B.nid 363941 zlib
/mnt/data/noaaport/nids/BGM/20220307_1929_N0B.nid 359692 zlib
/mnt/data/noaaport/nids/BGM/20220307_1935_N0B.nid 355491 zlib
/mnt/data/noaaport/nids/BGM/20220307_1941_N0B.nid 352040 zlib
/mnt/data/noaaport/nids/BGM/20220307_1947_N0B.nid 346301 zlib
/mnt/data/noaaport/nids/BGM/20220307_1953_N0B.nid 338393 zlib
/mnt/data/noaaport/nids/CBW/20220307_1916_N0B.nid 355862 zlib
/mnt/data/noaaport/nids/CBW/20220307_1922_N0B.nid 361885 zlib
/mnt/data/noaaport/nids/CBW/20220307_1928_N0B.nid 369223 zlib
/mnt/data/noaaport/nids/CBW/20220307_1934_N0B.nid 372702 zlib
/mnt/data/noaaport/nids/CBW/20220307_1940_N0B.nid 374620 zlib
/mnt/data/noaaport/nids/CBW/20220307_1946_N0B.nid 375270 zlib
/mnt/data/noaaport/nids/CBW/20220307_1952_N0B.nid 373890 zlib
/mnt/data/noaaport/nids/CCX/20220307_1901_N0B.nid 361946 zlib
/mnt/data/noaaport/nids/CCX/20220307_1907_N0B.nid 360498 zlib
/mnt/data/noaaport/nids/CCX/20220307_1913_N0B.nid 361337 zlib
/mnt/data/noaaport/nids/CCX/20220307_1918_N0B.nid 357527 zlib
/mnt/data/noaaport/nids/CCX/20220307_1924_N0B.nid 352904 zlib
/mnt/data/noaaport/nids/CCX/20220307_1930_N0B.nid 350048 zlib
/mnt/data/noaaport/nids/CCX/20220307_1936_N0B.nid 340482 zlib
/mnt/data/noaaport/nids/CCX/20220307_1942_N0B.nid 329713 zlib
/mnt/data/noaaport/nids/CCX/20220307_1948_N0B.nid 319300 zlib
/mnt/data/noaaport/nids/ENX/20220307_1912_N0B.nid 324750 zlib
/mnt/data/noaaport/nids/ENX/20220307_1930_N0B.nid 329362 zlib
/mnt/data/noaaport/nids/ENX/20220307_1941_N0B.nid 329805 zlib
/mnt/data/noaaport/nids/ENX/20220307_1947_N0B.nid 333013 zlib
/mnt/data/noaaport/nids/ENX/20220307_1953_N0B.nid 333542 zlib
/mnt/data/noaaport/nids/GYX/20220307_1940_N0B.nid 321435 zlib
/mnt/data/noaaport/nids/GYX/20220307_1946_N0B.nid 318901 zlib
/mnt/data/noaaport/nids/HTX/20220307_1902_N0B.nid 286756 zlib
/mnt/data/noaaport/nids/HTX/20220307_1905_N0B.nid 281872 zlib
/mnt/data/noaaport/nids/HTX/20220307_1912_N0B.nid 282532 zlib
/mnt/data/noaaport/nids/MRX/20220307_1902_N0B.nid 286064 zlib
/mnt/data/noaaport/nids/MRX/20220307_1908_N0B.nid 286126 zlib
/mnt/data/noaaport/nids/MRX/20220307_1914_N0B.nid 286212 zlib
/mnt/data/noaaport/nids/MRX/20220307_1920_N0B.nid 286076 zlib
/mnt/data/noaaport/nids/MRX/20220307_1926_N0B.nid 285670 zlib
/mnt/data/noaaport/nids/MRX/20220307_1932_N0B.nid 288133 zlib
/mnt/data/noaaport/nids/MRX/20220307_1938_N0B.nid 283514 zlib
/mnt/data/noaaport/nids/MRX/20220307_1944_N0B.nid 285975 zlib
/mnt/data/noaaport/nids/MRX/20220307_1950_N0B.nid 288229 zlib
/mnt/data/noaaport/nids/RLX/20220307_1901_N0B.nid 312651 zlib
/mnt/data/noaaport/nids/RLX/20220307_1903_N0B.nid 309328 zlib
/mnt/data/noaaport/nids/RLX/20220307_1905_N0B.nid 307597 zlib
/mnt/data/noaaport/nids/RLX/20220307_1911_N0B.nid 299850 zlib
/mnt/data/noaaport/nids/RLX/20220307_1918_N0B.nid 296619 zlib
/mnt/data/noaaport/nids/TYX/20220307_1936_N0B.nid 253444 zlib
Dan.
On Mon, Mar 7, 2022 at 1:32 PM Herzmann, Daryl E [AGRON] <
akrherz@xxxxxxxxxxx> wrote:
> Thanks Dan. I agree, my theory has been debunked!
>
> daryl
>
> ________________________________________
> From: Daniel Vietor - NOAA Affiliate <dan.vietor@xxxxxxxx>
> Sent: Monday, March 7, 2022 1:31 PM
> To: Mike Zuranski
> Cc: LDM-Users; Herzmann, Daryl E [AGRON]; Gilbert Sebenste; Mike Voss
> Subject: Re: [ldm-users] NEXRAD ID changes??
>
> Here is what I'm seeing:
>
> > pqlist -v -i 2 -p N0BHTX -O
> 282532 20220307/191247.959 NEXRAD3 73337671 SDUS54 KHUN 071912 /pN0BHTX
> desi.unidata.ucar.edu_v_noaaport2.wright-weather.com<
> http://desi.unidata.ucar.edu_v_noaaport2.wright-weather.com>
> 286260 20220307/191605.178 NEXRAD3 73347001 SDUS54 KHUN 071915 /pN0BHTX
> !nids/ desi.unidata.ucar.edu_v_noaaport2.wright-weather.com<
> http://desi.unidata.ucar.edu_v_noaaport2.wright-weather.com>
> 283571 20220307/191903.226 NEXRAD3 73355592 SDUS54 KHUN 071918 /pN0BHTX
> !nids/ desi.unidata.ucar.edu_v_noaaport2.wright-weather.com<
> http://desi.unidata.ucar.edu_v_noaaport2.wright-weather.com>
> 288726 20220307/192204.649 NEXRAD3 73363877 SDUS54 KHUN 071921 /pN0BHTX
> !nids/ desi.unidata.ucar.edu_v_noaaport2.wright-weather.com<
> http://desi.unidata.ucar.edu_v_noaaport2.wright-weather.com>
> 285988 20220307/192603.930 NEXRAD3 73374809 SDUS54 KHUN 071925 /pN0BHTX
> !nids/ desi.unidata.ucar.edu_v_noaaport2.wright-weather.com<
> http://desi.unidata.ucar.edu_v_noaaport2.wright-weather.com>
>
> desi.unidata seems to be the source for all the products. But those that
> are zlib compressed don't have a "!nids" on the end of the header,
>
> I checked our feed at AWC which is coming straight from a dish in the back
> 40.
>
> 00000000 01 0d 0d 0a 30 30 30 0d 0d 0a 53 44 55 53 35 34
> |....000...SDUS54|
> 00000010 20 4b 48 55 4e 20 30 37 31 39 31 32 0d 0d 0a 4e | KHUN
> 071912...N|
> 00000020 30 42 48 54 58 0d 0d 0a 78 da 85 57 7b 54 13 d7
> |0BHTX...x..W{T..|
> 00000030 ba 1f 1e 8a da 82 de ab d6 16 8a da f6 54 c4 9e
> |.............T..|
> 00000040 4a 55 1e 0a 26 82 68 7d b4 12 4f 11 53 c0 84 9e
> |JU..&.h}..O.S...|
> 00000050 52 e4 11 92 a8 90 cc 21 c3 58 d4 56 5b 2d a5 ad
> |R......!.X.V[-..|
> 00000060 0f 0a 31 80 72 25 2a 92 88 98 44 32 cc 5c 5b 7b
> |..1.r%*...D2.\[{|
> 00000070 44 94 80 8a 10 c2 30 33 ad 08 01 92 99 29 8f 64
> |D.....03.....).d|
>
> It's coming off the dish that way.
>
> Dan.
>
> On Mon, Mar 7, 2022 at 1:18 PM Mike Zuranski <zuranski.wx@xxxxxxxxx
> <mailto:zuranski.wx@xxxxxxxxx>> wrote:
> FWIW it's not just N.B products doing this...
>
> 20220307T191633.718220Z notifyme[70117]
> notifyme.c:notifymeprog_5:212 INFO 18370 20220307191632.821513
> NEXRAD3 73348475 SDUS26 KHNX 071911 /pNBBHNX !nids/
> 20220307T191633.718326Z notifyme[70117]
> notifyme.c:notifymeprog_5:212 INFO 6152 20220307191632.867833
> NEXRAD3 73348476 SDUS58 PAFC 071914 /pN0SAIH
> 20220307T191633.766274Z notifyme[70117]
> notifyme.c:notifymeprog_5:212 INFO 19778 20220307191632.870118
> NEXRAD3 73348477 SDUS28 PAFC 071910 /pN3BAKC !nids/
> 20220307T191633.766402Z notifyme[70117]
> notifyme.c:notifymeprog_5:212 INFO 2178 20220307191632.870344
> NEXRAD3 73348478 SDUS52 KMLB 071913 /pDPAMLB
> 20220307T191633.810195Z notifyme[70117]
> notifyme.c:notifymeprog_5:212 INFO 98306 20220307191632.886132
> NEXRAD3 73348479 SDUS21 KCTP 071913 /pN2BCCX !nids/
> 20220307T191633.810287Z notifyme[70117]
> notifyme.c:notifymeprog_5:212 INFO 1552 20220307191632.888075
> NEXRAD3 73348480 SDUS55 KSLC 071910 /pNCRSLC
> 20220307T191633.854166Z notifyme[70117]
> notifyme.c:notifymeprog_5:212 INFO 67908 20220307191633.029729
> NEXRAD3 73348481 SDUS52 KFFC 071915 /pTV0ATL !nids/
> 20220307T191633.854309Z notifyme[70117]
> notifyme.c:notifymeprog_5:212 INFO 4380 20220307191633.030216
> NEXRAD3 73348482 SDUS52 KMLB 071913 /pNTPMLB
> 20220307T191633.898156Z notifyme[70117]
> notifyme.c:notifymeprog_5:212 INFO 233 20220307191633.030298
> NEXRAD3 73348483 SDUS55 KSLC 071910 /pNVLSLC
> 20220307T191633.898265Z notifyme[70117]
> notifyme.c:notifymeprog_5:212 INFO 4050 20220307191633.032406
> NEXRAD3 73348484 SDUS52 KMLB 071913 /pDSPMLB !nids/
> 20220307T191633.946219Z notifyme[70117]
> notifyme.c:notifymeprog_5:212 INFO 4484 20220307191633.035764
> NEXRAD3 73348485 SDUS32 KMLB 071913 /pN1PMLB
> 20220307T191633.946316Z notifyme[70117]
> notifyme.c:notifymeprog_5:212 INFO 73239 20220307191633.047202
> NEXRAD3 73348486 SDUS55 KSLC 071915 /pNYGICX !nids/
> 20220307T191633.990205Z notifyme[70117]
> notifyme.c:notifymeprog_5:212 INFO 15915 20220307191633.049020
> NEXRAD3 73348487 SDUS26 KHNX 071911 /pNBUHNX !nids/
>
> I have not dug any deeper on those files however.
>
> -Mike
>
> ======================
> Mike Zuranski
> Meteorology Support Analyst
> College of DuPage - Nexlab
> Weather.cod.edu<http://weather.cod.edu/>
> ======================
>
>
> On Mon, Mar 7, 2022 at 1:11 PM Mike Zuranski <zuranski.wx@xxxxxxxxx
> <mailto:zuranski.wx@xxxxxxxxx>> wrote:
> I'm gonna go out on a limb and say it's a bug...
>
> An observation I made that correlates with yours: Shortly after RAX began
> issuing N.B products there was a period where they were coming in... badly
> somehow. The detail I noticed was the missing "!nids/" at the end of
> nexrad products while watching the nexrad3 feed. And like you noticed that
> was temporary.
>
> I see the same thing this morning from a number of sites (LOT grabbed my
> attention first). I also see the number of sites doing this are
> decreasing, so I'm wondering if this is being worked on somewhere. But any
> nexrad products that I see in LDM that do not contain "!nids/" at the end
> do not jive with McIDAS nor Gempak while all others do.
>
> Brief example:
> 20220307T185809.320700Z notifyme[10618]
> notifyme.c:notifymeprog_5:212 INFO 123297 20220307185809.170361
> NEXRAD3 73287741 SDUS58 PAFC 071856 /pN0BAKC !nids/
> 20220307T185811.881051Z notifyme[10618]
> notifyme.c:notifymeprog_5:212 INFO 314053 20220307185811.690988
> NEXRAD3 73287963 SDUS51 KRLX 071857 /pN0BRLX
> 20220307T185813.488004Z notifyme[10618]
> notifyme.c:notifymeprog_5:212 INFO 191116 20220307185813.222289
> NEXRAD3 73287985 SDUS53 KGRR 071856 /pN0BGRR !nids/
>
> Daryl: I'm not sure there's a correlation to exengine4.fox.com<
> http://exengine4.fox.com>, last time it was data straight out of SBN
> doing this. My Noaaport's been down a while so I can't confirm that right
> now though.
>
> -Mike
>
> ======================
> Mike Zuranski
> Meteorology Support Analyst
> College of DuPage - Nexlab
> Weather.cod.edu<http://weather.cod.edu/>
> ======================
>
>
> On Mon, Mar 7, 2022 at 1:04 PM Daniel Vietor - NOAA Affiliate via
> ldm-users <ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx>>
> wrote:
> Is it just me or are people seeing a strange zlib compression of the new
> N0B products. When RAX first came out, it was OK and then for a couple of
> days I saw the front end of the product was zlib compressed like the old
> days of the nexrad products. It went away so I didn't think much about it .
>
> But then yesterday it came back. Several sites were intermittently zlib
> compressing the N0B product. I noted Topeka, Kansas City, Springfield, St
> Louis, Chicago, Evansville, Indianapolis and N Indiana were doing it. But
> it was intermittent. One scan would be zlib compressed and the next
> wouldn't be. It was like something in the product, like size, was
> triggering the zlib compression.
>
> I took a look at some of the data. Only the first 4000 bytes are zlib
> compressed. The radial payload is still bzip2 compressed. Since only the
> header, which is about 100 bytes, is uncompressed, the zlib compression is
> compressing over 3800 bytes of the radial payload. This seems strange. Why
> compress data that are already compressed?
>
> After last night, it seems like this is a feature of the new N0B radar
> data and not a bug. So I resurrected the old ucnid.c program I wrote 24
> years ago to remove the zlib compression on the N0B products.
>
> So is this a bug or a feature?
>
> Dan.
>
> On Fri, Mar 4, 2022 at 10:36 AM Gilbert Sebenste <gilbert@xxxxxxxxxxxxxxxx
> <mailto:gilbert@xxxxxxxxxxxxxxxx>> wrote:
> Nothing beats a good Daryl LDM-users rant in the morning. He never lets me
> down!
>
> Imagine my utter surprise when I did what Daryl said and I found products
> I didn't know were being sent over NOAAport. Then, I got curious and
> instead of using the "NEXRAD" feedtype, I used "ANY". That's when I was
> shocked to find several products being sent under the HRS feed that were
> completely undocumented by the NWS, and were needed! I alerted Steve
> Emmerson, who put out a fix for it. 6.13.16 has it all sorted out now.
>
> I'll point out that this thinking goes for any product. If there's a
> product you think should be there that isn't coming in, and you know you're
> getting the entire feed, use a request of "ANY" on that header. Rarely,
> it's an LDM bug, but like the example above, it can happen where a product
> shows up on an unexpected feedtype.
>
> Gilbert
>
> > On Mar 4, 2022, at 10:17 AM, Herzmann, Daryl E [AGRON] <
> akrherz@xxxxxxxxxxx<mailto:akrherz@xxxxxxxxxxx>> wrote:
> >
> > Good morning,
> >
> > I'd like to chime on the topic of "best practice for pqact entries".
> For the WMO header found in products, there's a general form for the
> entries found:
> >
> > TTAAII CCCC DDHHMI
> >
> > When there's a string found in the line below the WMO header that is six
> or so characters, this "PIL" is appended to LDM product name in the form
> like so:
> >
> > TTAAII CCCC DDHHMI /pPILXXX
> >
> > Summary: I *do not* trust the TTAAII value and only sparingly use it
> within a pqact entry, but instead fully trust the /pPILXXX section.
> >
> > In December 2015, I meet with the NWS Data Management folks expressing
> interest in quality controlling the CCCC portion of the WMO header. Can
> you imagine that it is sometimes wrong? Well, it has been 6+ years now,
> hundreds of emails, and I'm still not done getting them to correct issues
> with it. As an example of how painful the rabbit hole is, NWS Cheyenne
> KCYS was issuing TAFs for 3 sites using KOAX as the CCCC. This took 8
> months to fix and I am still unsure if it is fully fixed.
> >
> > I have not even attempted automated quality control of the TTAAII, nor
> bugging NWS to fix it. Another story. I actually did attempt to get NWS
> to fix the TTAAII in the case of a product that wasn't using 6 characters
> for the TTAAII, can you imagine that it is sometimes only 5? That request
> was rejected. Anyway, this value used to be more rigorously set, but is
> now more arbitrary than ever. So in the original example below.
> >
> > NEXRAD ^SDUS[2357]. ....
> ([0-3][0-9])([0-2][0-9])([0-6][0-9]).*/p(...)(...)
> >
> > I would only trust the SDUS and drop any checks on the other TTAAII
> fields. Instead, put your limiters into the /p section.
> >
> > NEXRAD ^SDUS.. ....
> ([0-3][0-9])([0-2][0-9])([0-6][0-9]).*/p(N0Q|N0B|N0G|Nwhatever)(...)
> >
> > Oh, there are examples in the TTAAII, where the AA is wrong, but I have
> ranted/whined enough already.
> >
> > daryl
> >
> > ________________________________________
> > From: ldm-users <ldm-users-bounces@xxxxxxxxxxxxxxxx<mailto:
> ldm-users-bounces@xxxxxxxxxxxxxxxx>> on behalf of Gilbert Sebenste <
> gilbert@xxxxxxxxxxxxxxxx<mailto:gilbert@xxxxxxxxxxxxxxxx>>
> > Sent: Friday, March 4, 2022 9:54 AM
> > To: Daniel Vietor - NOAA Affiliate
> > Cc: LDM-Users; Mike Voss
> > Subject: Re: [ldm-users] NEXRAD ID changes??
> >
> > If you have LDM version 6.13.16, this shouldn't be happening. A bug fix
> was applied to ensure that everything NEXRAD3 is supposed to be there.
> >
> > Gilbert
> >
> > On Mar 4, 2022, at 9:15 AM, Daniel Vietor - NOAA Affiliate via ldm-users
> <ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx>> wrote:
> >
> >
> > I see a lot of them on the HDS feedtype. So I request both NEXRAD and
> HDS.
> >
> > Dan.
> >
> > On Fri, Mar 4, 2022 at 9:04 AM Mike Voss <mike.voss@xxxxxxxx<mailto:
> mike.voss@xxxxxxxx><mailto:mike.voss@xxxxxxxx<mailto:mike.voss@xxxxxxxx>>>
> wrote:
> > Hi All,
> > For me at least, some of my NEXRAD products stopped filing correctly
> yesterday at 18Z using this generic pattern match:
> >
> > NEXRAD ^SDUS[2357]. ....
> ([0-3][0-9])([0-2][0-9])([0-6][0-9]).*/p(...)(...)
> > FILE -overwrite -close
> nexrad/NIDS/\5/\4/\4_(\1:yyyy)(\1:mm)\1_\2\3
> >
> > as a test, when I changed to this:
> > NEXRAD ^SDUS66 ....
> >
> > I started getting my "N0Q" products for MUX station.
> >
> > Also, I stopped receiving the FNEXRAD NEXRCOMP products. Is it possible
> these things are related such that the NEXRCOMP depends on the NIDS product
> ID's and somehow a recent change (yesterday) cause this to break?
> >
> > thanks for any insights,
> > -MIke
> > _______________________________________________
> > NOTE: All exchanges posted to Unidata maintained email lists are
> > recorded in the Unidata inquiry tracking system and made publicly
> > available through the web. Users who post to any of the lists we
> > maintain are reminded to remove any personal information that they
> > do not want to be made public.
> >
> >
> > ldm-users mailing list
> > ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx><mailto:
> ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx>>
> > For list information or to unsubscribe, visit:
> https://www.unidata.ucar.edu/mailing_lists/
> >
> >
> > --
> > Dan Vietor
> > Senior Research Meteorologist
> > CIRA, Colorado State Univ
> > Aviation Weather Center
> > Kansas City, MO
> > 816.584.7211
> >
> > _______________________________________________
> > NOTE: All exchanges posted to Unidata maintained email lists are
> > recorded in the Unidata inquiry tracking system and made publicly
> > available through the web. Users who post to any of the lists we
> > maintain are reminded to remove any personal information that they
> > do not want to be made public.
> >
> >
> > ldm-users mailing list
> > ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx>
> > For list information or to unsubscribe, visit:
> https://www.unidata.ucar.edu/mailing_lists/
>
>
> --
> Dan Vietor
> Senior Research Meteorologist
> CIRA, Colorado State Univ
> Aviation Weather Center
> Kansas City, MO
> 816.584.7211
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> ldm-users mailing list
> ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx>
> For list information or to unsubscribe, visit:
> https://www.unidata.ucar.edu/mailing_lists/
>
>
> --
> Dan Vietor
> Senior Research Meteorologist
> CIRA, Colorado State Univ
> Aviation Weather Center
> Kansas City, MO
> 816.584.7211
>
>
--
*Dan Vietor*
*Senior Research Meteorologist*
CIRA, Colorado State Univ
Aviation Weather Center
Kansas City, MO
816.584.7211
