Greetings Dan,
A current working theory is that these are coming from the infamous
wxengine4.fox.com ingest. :) If you can send me exact N0B file names with the
problem, I can look them up in my IDD logging and see which noaaport ingest
source they came from.
daryl
________________________________________
From: Daniel Vietor - NOAA Affiliate <dan.vietor@xxxxxxxx>
Sent: Monday, March 7, 2022 1:04 PM
To: Gilbert Sebenste
Cc: Herzmann, Daryl E [AGRON]; LDM-Users; Mike Voss
Subject: Re: [ldm-users] NEXRAD ID changes??
Is it just me or are people seeing a strange zlib compression of the new N0B
products. When RAX first came out, it was OK and then for a couple of days I
saw the front end of the product was zlib compressed like the old days of the
nexrad products. It went away so I didn't think much about it.
But then yesterday it came back. Several sites were intermittently zlib
compressing the N0B product. I noted Topeka, Kansas City, Springfield, St
Louis, Chicago, Evansville, Indianapolis and N Indiana were doing it. But it
was intermittent. One scan would be zlib compressed and the next wouldn't be.
It was like something in the product, like size, was triggering the zlib
compression.
I took a look at some of the data. Only the first 4000 bytes are zlib
compressed. The radial payload is still bzip2 compressed. Since only the
header, which is about 100 bytes, is uncompressed, the zlib compression is
compressing over 3800 bytes of the radial payload. This seems strange. Why
compress data that are already compressed?
After last night, it seems like this is a feature of the new N0B radar data and
not a bug. So I resurrected the old ucnid.c program I wrote 24 years ago to
remove the zlib compression on the N0B products.
So is this a bug or a feature?
Dan.
On Fri, Mar 4, 2022 at 10:36 AM Gilbert Sebenste
<gilbert@xxxxxxxxxxxxxxxx<mailto:gilbert@xxxxxxxxxxxxxxxx>> wrote:
Nothing beats a good Daryl LDM-users rant in the morning. He never lets me down!
Imagine my utter surprise when I did what Daryl said and I found products I
didn't know were being sent over NOAAport. Then, I got curious and instead of
using the "NEXRAD" feedtype, I used "ANY". That's when I was shocked to find
several products being sent under the HRS feed that were completely
undocumented by the NWS, and were needed! I alerted Steve Emmerson, who put out
a fix for it. 6.13.16 has it all sorted out now.
I'll point out that this thinking goes for any product. If there's a product
you think should be there that isn't coming in, and you know you're getting the
entire feed, use a request of "ANY" on that header. Rarely, it's an LDM bug,
but like the example above, it can happen where a product shows up on an
unexpected feedtype.
Gilbert
> On Mar 4, 2022, at 10:17 AM, Herzmann, Daryl E [AGRON]
> <akrherz@xxxxxxxxxxx<mailto:akrherz@xxxxxxxxxxx>> wrote:
>
> Good morning,
>
> I'd like to chime on the topic of "best practice for pqact entries". For the
> WMO header found in products, there's a general form for the entries found:
>
> TTAAII CCCC DDHHMI
>
> When there's a string found in the line below the WMO header that is six or
> so characters, this "PIL" is appended to LDM product name in the form like so:
>
> TTAAII CCCC DDHHMI /pPILXXX
>
> Summary: I *do not* trust the TTAAII value and only sparingly use it within a
> pqact entry, but instead fully trust the /pPILXXX section.
>
> In December 2015, I meet with the NWS Data Management folks expressing
> interest in quality controlling the CCCC portion of the WMO header. Can you
> imagine that it is sometimes wrong? Well, it has been 6+ years now, hundreds
> of emails, and I'm still not done getting them to correct issues with it. As
> an example of how painful the rabbit hole is, NWS Cheyenne KCYS was issuing
> TAFs for 3 sites using KOAX as the CCCC. This took 8 months to fix and I am
> still unsure if it is fully fixed.
>
> I have not even attempted automated quality control of the TTAAII, nor
> bugging NWS to fix it. Another story. I actually did attempt to get NWS to
> fix the TTAAII in the case of a product that wasn't using 6 characters for
> the TTAAII, can you imagine that it is sometimes only 5? That request was
> rejected. Anyway, this value used to be more rigorously set, but is now
> more arbitrary than ever. So in the original example below.
>
> NEXRAD ^SDUS[2357]. .... ([0-3][0-9])([0-2][0-9])([0-6][0-9]).*/p(...)(...)
>
> I would only trust the SDUS and drop any checks on the other TTAAII fields.
> Instead, put your limiters into the /p section.
>
> NEXRAD ^SDUS.. ....
> ([0-3][0-9])([0-2][0-9])([0-6][0-9]).*/p(N0Q|N0B|N0G|Nwhatever)(...)
>
> Oh, there are examples in the TTAAII, where the AA is wrong, but I have
> ranted/whined enough already.
>
> daryl
>
> ________________________________________
> From: ldm-users
> <ldm-users-bounces@xxxxxxxxxxxxxxxx<mailto:ldm-users-bounces@xxxxxxxxxxxxxxxx>>
> on behalf of Gilbert Sebenste
> <gilbert@xxxxxxxxxxxxxxxx<mailto:gilbert@xxxxxxxxxxxxxxxx>>
> Sent: Friday, March 4, 2022 9:54 AM
> To: Daniel Vietor - NOAA Affiliate
> Cc: LDM-Users; Mike Voss
> Subject: Re: [ldm-users] NEXRAD ID changes??
>
> If you have LDM version 6.13.16, this shouldn't be happening. A bug fix was
> applied to ensure that everything NEXRAD3 is supposed to be there.
>
> Gilbert
>
> On Mar 4, 2022, at 9:15 AM, Daniel Vietor - NOAA Affiliate via ldm-users
> <ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx>> wrote:
>
>
> I see a lot of them on the HDS feedtype. So I request both NEXRAD and HDS.
>
> Dan.
>
> On Fri, Mar 4, 2022 at 9:04 AM Mike Voss
> <mike.voss@xxxxxxxx<mailto:mike.voss@xxxxxxxx><mailto:mike.voss@xxxxxxxx<mailto:mike.voss@xxxxxxxx>>>
> wrote:
> Hi All,
> For me at least, some of my NEXRAD products stopped filing correctly
> yesterday at 18Z using this generic pattern match:
>
> NEXRAD ^SDUS[2357]. .... ([0-3][0-9])([0-2][0-9])([0-6][0-9]).*/p(...)(...)
> FILE -overwrite -close
> nexrad/NIDS/\5/\4/\4_(\1:yyyy)(\1:mm)\1_\2\3
>
> as a test, when I changed to this:
> NEXRAD ^SDUS66 ....
>
> I started getting my "N0Q" products for MUX station.
>
> Also, I stopped receiving the FNEXRAD NEXRCOMP products. Is it possible
> these things are related such that the NEXRCOMP depends on the NIDS product
> ID's and somehow a recent change (yesterday) cause this to break?
>
> thanks for any insights,
> -MIke
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> ldm-users mailing list
> ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx><mailto:ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx>>
> For list information or to unsubscribe, visit:
> https://www.unidata.ucar.edu/mailing_lists/
>
>
> --
> Dan Vietor
> Senior Research Meteorologist
> CIRA, Colorado State Univ
> Aviation Weather Center
> Kansas City, MO
> 816.584.7211
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> ldm-users mailing list
> ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx>
> For list information or to unsubscribe, visit:
> https://www.unidata.ucar.edu/mailing_lists/
--
Dan Vietor
Senior Research Meteorologist
CIRA, Colorado State Univ
Aviation Weather Center
Kansas City, MO
816.584.7211
