HTX is showing it now. The N0B files from 1830 to 1902 are all zlib
compressed. The 1908 product is not.
Here is the header for HTX/20220307_1902_N0B.nid
00000000 01 0d 0d 0a 38 35 33 20 0d 0d 0a 53 44 55 53 35 |....853
...SDUS5|
00000010 34 20 4b 48 55 4e 20 30 37 31 39 30 32 0d 0d 0a |4 KHUN
071902...|
00000020 4e 30 42 48 54 58 0d 0d 0a *78 da *85 57 7d 54 53
|N0BHTX...x..W}TS|
00000030 57 b6 bf 7c 54 a7 b5 68 3f c7 56 0b 68 a7 4f 5d
|W..|T..h?.V.h.O]|
00000040 b5 7e 55 84 2a 26 a5 b5 83 1d e5 a3 23 22 42 9a
|.~U.*&......#"B.|
00000050 60 47 41 21 92 a8 31 b9 92 eb ed 50 9d 0e d3 0f
|`GA!..1....P....|
00000060 75 5a 46 ad 50 3e 5a 2d 08 98 e4 d5 98 5c e1 72
|uZF.P>Z-.....\.r|
00000070 ee 73 6c 2b 4a 25 01 23 84 70 73 6f 56 ad 12 f0
|.sl+J%.#.psoV...|
The 78 da bytes show the product is zlib compressed.
Here is the 1908 product:
00000000 01 0d 0d 0a 33 36 33 20 0d 0d 0a 53 44 55 53 35 |....363
...SDUS5|
00000010 34 20 4b 48 55 4e 20 30 37 31 39 30 38 0d 0d 0a |4 KHUN
071908...|
00000020 4e 30 42 48 54 58 0d 0d 0a 00 99 4a 73 00 01 0d
|N0BHTX.....Js...|
00000030 6c 00 04 56 79 03 3a 00 00 00 03 ff ff 00 00 88
|l..Vy.:.........|
00000040 73 ff fe af bc 07 43 00 99 00 02 00 d7 13 24 00
|s.....C.......$.|
00000050 0b 4a 73 00 01 0d 4b 4a 73 00 01 0d 6c 00 00 00
|.Js...KJs...l...|
00000060 00 00 07 00 05 fe c0 00 05 00 fe 00 00 00 00 00
|................|
00000070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
00000080 00 00 00 00 00 00 3c 00 00 00 00 1e 82 00 01 00
|......<.........|
00000090 14 47 fe 00 00 00 00 00 3c 00 00 00 00 00 00 00
|.G......<.......|
000000a0 00 42 5a 68 39 31 41 59 26 53 59 29 d7 03 52 03 |.*BZ*
h91AY&SY)..R.|
The BZ is the start of the bzip2 compression. Header is uncompressed so you
can see the BZ fingerprint.
Dan.
On Mon, Mar 7, 2022 at 1:12 PM Mike Zuranski <zuranski.wx@xxxxxxxxx> wrote:
> I'm gonna go out on a limb and say it's a bug...
>
> An observation I made that correlates with yours: Shortly after RAX began
> issuing N.B products there was a period where they were coming in... badly
> somehow. The detail I noticed was the missing "!nids/" at the end of
> nexrad products while watching the nexrad3 feed. And like you noticed that
> was temporary.
>
> I see the same thing this morning from a number of sites (LOT grabbed my
> attention first). I also see the number of sites doing this are
> decreasing, so I'm wondering if this is being worked on somewhere. But any
> nexrad products that I see in LDM that do not contain "!nids/" at the end
> do not jive with McIDAS nor Gempak while all others do.
>
> Brief example:
> 20220307T185809.320700Z notifyme[10618]
> notifyme.c:notifymeprog_5:212 INFO 123297 20220307185809.170361
> NEXRAD3 73287741 SDUS58 PAFC 071856 /pN0BAKC !nids/
> 20220307T185811.881051Z notifyme[10618]
> notifyme.c:notifymeprog_5:212 INFO 314053 20220307185811.690988
> NEXRAD3 73287963 SDUS51 KRLX 071857 /pN0BRLX
> 20220307T185813.488004Z notifyme[10618]
> notifyme.c:notifymeprog_5:212 INFO 191116 20220307185813.222289
> NEXRAD3 73287985 SDUS53 KGRR 071856 /pN0BGRR !nids/
>
> Daryl: I'm not sure there's a correlation to exengine4.fox.com, last
> time it was data straight out of SBN doing this. My Noaaport's been down a
> while so I can't confirm that right now though.
>
> -Mike
>
> ======================
> Mike Zuranski
> Meteorology Support Analyst
> College of DuPage - Nexlab
> Weather.cod.edu <http://weather.cod.edu/>
> ======================
>
>
> On Mon, Mar 7, 2022 at 1:04 PM Daniel Vietor - NOAA Affiliate via
> ldm-users <ldm-users@xxxxxxxxxxxxxxxx> wrote:
>
>> Is it just me or are people seeing a strange zlib compression of the new
>> N0B products. When RAX first came out, it was OK and then for a couple of
>> days I saw the front end of the product was zlib compressed like the old
>> days of the nexrad products. It went away so I didn't think much about it.
>>
>> But then yesterday it came back. Several sites were intermittently zlib
>> compressing the N0B product. I noted Topeka, Kansas City, Springfield, St
>> Louis, Chicago, Evansville, Indianapolis and N Indiana were doing it. But
>> it was intermittent. One scan would be zlib compressed and the next
>> wouldn't be. It was like something in the product, like size, was
>> triggering the zlib compression.
>>
>> I took a look at some of the data. Only the first 4000 bytes are zlib
>> compressed. The radial payload is still bzip2 compressed. Since only the
>> header, which is about 100 bytes, is uncompressed, the zlib compression is
>> compressing over 3800 bytes of the radial payload. This seems strange. Why
>> compress data that are already compressed?
>>
>> After last night, it seems like this is a feature of the new N0B radar
>> data and not a bug. So I resurrected the old ucnid.c program I wrote 24
>> years ago to remove the zlib compression on the N0B products.
>>
>> So is this a bug or a feature?
>>
>> Dan.
>>
>> On Fri, Mar 4, 2022 at 10:36 AM Gilbert Sebenste <
>> gilbert@xxxxxxxxxxxxxxxx> wrote:
>>
>>> Nothing beats a good Daryl LDM-users rant in the morning. He never lets
>>> me down!
>>>
>>> Imagine my utter surprise when I did what Daryl said and I found
>>> products I didn't know were being sent over NOAAport. Then, I got curious
>>> and instead of using the "NEXRAD" feedtype, I used "ANY". That's when I was
>>> shocked to find several products being sent under the HRS feed that were
>>> completely undocumented by the NWS, and were needed! I alerted Steve
>>> Emmerson, who put out a fix for it. 6.13.16 has it all sorted out now.
>>>
>>> I'll point out that this thinking goes for any product. If there's a
>>> product you think should be there that isn't coming in, and you know you're
>>> getting the entire feed, use a request of "ANY" on that header. Rarely,
>>> it's an LDM bug, but like the example above, it can happen where a product
>>> shows up on an unexpected feedtype.
>>>
>>> Gilbert
>>>
>>> > On Mar 4, 2022, at 10:17 AM, Herzmann, Daryl E [AGRON] <
>>> akrherz@xxxxxxxxxxx> wrote:
>>> >
>>> > Good morning,
>>> >
>>> > I'd like to chime on the topic of "best practice for pqact entries".
>>> For the WMO header found in products, there's a general form for the
>>> entries found:
>>> >
>>> > TTAAII CCCC DDHHMI
>>> >
>>> > When there's a string found in the line below the WMO header that is
>>> six or so characters, this "PIL" is appended to LDM product name in the
>>> form like so:
>>> >
>>> > TTAAII CCCC DDHHMI /pPILXXX
>>> >
>>> > Summary: I *do not* trust the TTAAII value and only sparingly use it
>>> within a pqact entry, but instead fully trust the /pPILXXX section.
>>> >
>>> > In December 2015, I meet with the NWS Data Management folks expressing
>>> interest in quality controlling the CCCC portion of the WMO header. Can
>>> you imagine that it is sometimes wrong? Well, it has been 6+ years now,
>>> hundreds of emails, and I'm still not done getting them to correct issues
>>> with it. As an example of how painful the rabbit hole is, NWS Cheyenne
>>> KCYS was issuing TAFs for 3 sites using KOAX as the CCCC. This took 8
>>> months to fix and I am still unsure if it is fully fixed.
>>> >
>>> > I have not even attempted automated quality control of the TTAAII, nor
>>> bugging NWS to fix it. Another story. I actually did attempt to get NWS
>>> to fix the TTAAII in the case of a product that wasn't using 6 characters
>>> for the TTAAII, can you imagine that it is sometimes only 5? That request
>>> was rejected. Anyway, this value used to be more rigorously set, but is
>>> now more arbitrary than ever. So in the original example below.
>>> >
>>> > NEXRAD ^SDUS[2357]. ....
>>> ([0-3][0-9])([0-2][0-9])([0-6][0-9]).*/p(...)(...)
>>> >
>>> > I would only trust the SDUS and drop any checks on the other TTAAII
>>> fields. Instead, put your limiters into the /p section.
>>> >
>>> > NEXRAD ^SDUS.. ....
>>> ([0-3][0-9])([0-2][0-9])([0-6][0-9]).*/p(N0Q|N0B|N0G|Nwhatever)(...)
>>> >
>>> > Oh, there are examples in the TTAAII, where the AA is wrong, but I
>>> have ranted/whined enough already.
>>> >
>>> > daryl
>>> >
>>> > ________________________________________
>>> > From: ldm-users <ldm-users-bounces@xxxxxxxxxxxxxxxx> on behalf of
>>> Gilbert Sebenste <gilbert@xxxxxxxxxxxxxxxx>
>>> > Sent: Friday, March 4, 2022 9:54 AM
>>> > To: Daniel Vietor - NOAA Affiliate
>>> > Cc: LDM-Users; Mike Voss
>>> > Subject: Re: [ldm-users] NEXRAD ID changes??
>>> >
>>> > If you have LDM version 6.13.16, this shouldn't be happening. A bug
>>> fix was applied to ensure that everything NEXRAD3 is supposed to be there.
>>> >
>>> > Gilbert
>>> >
>>> > On Mar 4, 2022, at 9:15 AM, Daniel Vietor - NOAA Affiliate via
>>> ldm-users <ldm-users@xxxxxxxxxxxxxxxx> wrote:
>>> >
>>> >
>>> > I see a lot of them on the HDS feedtype. So I request both NEXRAD and
>>> HDS.
>>> >
>>> > Dan.
>>> >
>>> > On Fri, Mar 4, 2022 at 9:04 AM Mike Voss <mike.voss@xxxxxxxx<mailto:
>>> mike.voss@xxxxxxxx>> wrote:
>>> > Hi All,
>>> > For me at least, some of my NEXRAD products stopped filing correctly
>>> yesterday at 18Z using this generic pattern match:
>>> >
>>> > NEXRAD ^SDUS[2357]. ....
>>> ([0-3][0-9])([0-2][0-9])([0-6][0-9]).*/p(...)(...)
>>> > FILE -overwrite -close
>>> nexrad/NIDS/\5/\4/\4_(\1:yyyy)(\1:mm)\1_\2\3
>>> >
>>> > as a test, when I changed to this:
>>> > NEXRAD ^SDUS66 ....
>>> >
>>> > I started getting my "N0Q" products for MUX station.
>>> >
>>> > Also, I stopped receiving the FNEXRAD NEXRCOMP products. Is it
>>> possible these things are related such that the NEXRCOMP depends on the
>>> NIDS product ID's and somehow a recent change (yesterday) cause this to
>>> break?
>>> >
>>> > thanks for any insights,
>>> > -MIke
>>> > _______________________________________________
>>> > NOTE: All exchanges posted to Unidata maintained email lists are
>>> > recorded in the Unidata inquiry tracking system and made publicly
>>> > available through the web. Users who post to any of the lists we
>>> > maintain are reminded to remove any personal information that they
>>> > do not want to be made public.
>>> >
>>> >
>>> > ldm-users mailing list
>>> > ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx>
>>> > For list information or to unsubscribe, visit:
>>> https://www.unidata.ucar.edu/mailing_lists/
>>> >
>>> >
>>> > --
>>> > Dan Vietor
>>> > Senior Research Meteorologist
>>> > CIRA, Colorado State Univ
>>> > Aviation Weather Center
>>> > Kansas City, MO
>>> > 816.584.7211
>>> >
>>> > _______________________________________________
>>> > NOTE: All exchanges posted to Unidata maintained email lists are
>>> > recorded in the Unidata inquiry tracking system and made publicly
>>> > available through the web. Users who post to any of the lists we
>>> > maintain are reminded to remove any personal information that they
>>> > do not want to be made public.
>>> >
>>> >
>>> > ldm-users mailing list
>>> > ldm-users@xxxxxxxxxxxxxxxx
>>> > For list information or to unsubscribe, visit:
>>> https://www.unidata.ucar.edu/mailing_lists/
>>>
>>
>>
>> --
>> *Dan Vietor*
>> *Senior Research Meteorologist*
>> CIRA, Colorado State Univ
>> Aviation Weather Center
>> Kansas City, MO
>> 816.584.7211
>>
>> _______________________________________________
>> NOTE: All exchanges posted to Unidata maintained email lists are
>> recorded in the Unidata inquiry tracking system and made publicly
>> available through the web. Users who post to any of the lists we
>> maintain are reminded to remove any personal information that they
>> do not want to be made public.
>>
>>
>> ldm-users mailing list
>> ldm-users@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> https://www.unidata.ucar.edu/mailing_lists/
>>
>
--
*Dan Vietor*
*Senior Research Meteorologist*
CIRA, Colorado State Univ
Aviation Weather Center
Kansas City, MO
816.584.7211
