Stonie,
I sent the example from the IDD telemetry logs I keep:
wmo_valid_at | product_origin |
seqnum | size
------------------------+-----------------------------------------------+----------+------
2023-03-29 13:54:00+00 | exo.aggregatehail.com_v_idd.agron.iastate.edu |
999 | 742 <-- here
2023-03-29 13:54:00+00 | np2.ssec.wisc.edu_v_idd.agron.iastate.edu |
73462429 | 743
2023-03-29 14:07:00+00 | exo.aggregatehail.com_v_idd.agron.iastate.edu |
999 | 742 <-- here
2023-03-29 14:07:00+00 | leno.unidata.ucar.edu_v_idd.agron.iastate.edu |
73472992 | 743
The one byte difference is due to the missing space after the 999 sequence
number, so LDM computes different MD5s after ignoring the first 11 bytes and
thus does not reject the products as duplicates.
daryl
________________________________________
From: Stonie Cooper <cooper@xxxxxxxx>
Sent: Wednesday, March 29, 2023 10:16 AM
To: Herzmann, Daryl E [AGRON]
Cc: Sebenste, Gilbert; Mike Zuranski; LDM; NOAAPORT
Subject: Re: [ldm-users] [External] Re: [noaaport] NOAAport/SBN outage as of
13:30Z
I'm a bit mystified by your request . . . I just illustrated that the
duplicates inserted from the script were deduplicated and not sent forward,
regardless of the space or lack of space, . . . is the issue you see downstream
from there?
+----------------------------------------------------------------+
* Stonie Cooper UCAR Unidata Program *
* Software Engineer III P.O. Box 3000 *
* cooper@xxxxxxxx<mailto:cooper@xxxxxxxx> Boulder,
CO 80307 *
* Unidata WWW Service http://www.unidata.ucar.edu/ *
+----------------------------------------------------------------+
On Wed, Mar 29, 2023 at 10:10 AM Herzmann, Daryl E [AGRON]
<akrherz@xxxxxxxxxxx<mailto:akrherz@xxxxxxxxxxx>> wrote:
Stonie,
Good grief, then fix LDM then to stop assuming 11 bytes to ignore and make it
forgive the lack of a space. Why punt this issue to every IDD downstream user,
when it can be fixed it at the source?
daryl
________________________________________
From: Stonie Cooper <cooper@xxxxxxxx<mailto:cooper@xxxxxxxx>>
Sent: Wednesday, March 29, 2023 10:06 AM
To: Herzmann, Daryl E [AGRON]
Cc: Sebenste, Gilbert; Mike Zuranski; LDM; NOAAPORT
Subject: Re: [ldm-users] [External] Re: [noaaport] NOAAport/SBN outage as of
13:30Z
I'd also like to see the ICD that says there is supposed to be a space after
the artificially applied sequence number. I can relate equal numbers of met
data users that complain about the existence or absence of this particular
Alden artifact, when in the end, just make your decoder resilient to accept
either.
+----------------------------------------------------------------+
* Stonie Cooper UCAR Unidata Program *
* Software Engineer III P.O. Box 3000 *
*
cooper@xxxxxxxx<mailto:cooper@xxxxxxxx><mailto:cooper@xxxxxxxx<mailto:cooper@xxxxxxxx>>
Boulder, CO 80307 *
* Unidata WWW Service http://www.unidata.ucar.edu/ *
+----------------------------------------------------------------+
On Wed, Mar 29, 2023 at 10:00 AM Herzmann, Daryl E [AGRON]
<akrherz@xxxxxxxxxxx<mailto:akrherz@xxxxxxxxxxx><mailto:akrherz@xxxxxxxxxxx<mailto:akrherz@xxxxxxxxxxx>>>
wrote:
Howdy Gilbert,
I'm gonna sound like a grump again, but we need to be very careful about
putting EMWIN data onto the IDD feed. The reason is that the EMWIN and
NOAAPort feeds are not byte-by-byte identical, so LDM has no chance to properly
dedup these products.
For example, the ADMSDM product was duplicated twice:
wmo_valid_at | product_origin |
seqnum | size
------------------------+-----------------------------------------------+----------+------
2023-03-29 13:54:00+00 |
exo.aggregatehail.com_v_idd.agron.iastate.edu<http://exo.aggregatehail.com_v_idd.agron.iastate.edu><http://exo.aggregatehail.com_v_idd.agron.iastate.edu>
| 999 | 742
2023-03-29 13:54:00+00 |
np2.ssec.wisc.edu_v_idd.agron.iastate.edu<http://np2.ssec.wisc.edu_v_idd.agron.iastate.edu><http://np2.ssec.wisc.edu_v_idd.agron.iastate.edu>
| 73462429 | 743
2023-03-29 14:07:00+00 |
exo.aggregatehail.com_v_idd.agron.iastate.edu<http://exo.aggregatehail.com_v_idd.agron.iastate.edu><http://exo.aggregatehail.com_v_idd.agron.iastate.edu>
| 999 | 742
2023-03-29 14:07:00+00 |
leno.unidata.ucar.edu_v_idd.agron.iastate.edu<http://leno.unidata.ucar.edu_v_idd.agron.iastate.edu><http://leno.unidata.ucar.edu_v_idd.agron.iastate.edu>
| 73472992 | 743
(4 rows)
As a reminder, the LDM MD5 checksum ignores the first 11 bytes of a product, so
the "sequence number" at the top of the product does not matter for the MD5
calculation. The most common difference between the two feeds is the presence
or absence of CR or LF at the end of the product, but this does not account for
all of the differences. Another difference is the "Aviation Control Character"
that gets stripped out.
In this case, and importantly to whomever wrote the EMWIN to IDD ingest, is the
missing space character after the sequence number:
999^M^M <-- missing space here
NOUS42 KWNO 291407^M^M
ADMSDM^M^M
This space needs to be there for the 11 byte ignore calculation to be correct.
It would be nice that we could turn on this level of redundancy full-time, but
we'd likely need to condition the noaaport + EMWIN ingests to rectify these
differences so that LDM can generate identical MD5 and properly dedup the feeds.
daryl
________________________________________
From: ldm-users
<ldm-users-bounces@xxxxxxxxxxxxxxxx<mailto:ldm-users-bounces@xxxxxxxxxxxxxxxx><mailto:ldm-users-bounces@xxxxxxxxxxxxxxxx<mailto:ldm-users-bounces@xxxxxxxxxxxxxxxx>>>
on behalf of Sebenste, Gilbert
<sebensteg@xxxxxxx<mailto:sebensteg@xxxxxxx><mailto:sebensteg@xxxxxxx<mailto:sebensteg@xxxxxxx>>>
Sent: Wednesday, March 29, 2023 9:18 AM
To: Mike Zuranski
Cc: LDM; NOAAPORT
Subject: Re: [ldm-users] [External] Re: [noaaport] NOAAport/SBN outage as of
13:30Z
AllisonHouse just turned on a kludged EMWIN feed to at least get some
IDS|DDPLUS products going again until this outage is over with. They are
sending it to UNIDATA, so they hope it is getting through the system.
Gilbert Sebenste
Meteorology Support Analyst
[cid:image001.png@01D9621F.5C34C250]
From: Mike Zuranski
<mzuranski@xxxxxxxx<mailto:mzuranski@xxxxxxxx><mailto:mzuranski@xxxxxxxx<mailto:mzuranski@xxxxxxxx>>>
Sent: Wednesday, March 29, 2023 8:44 AM
To: Sebenste, Gilbert
<sebensteg@xxxxxxx<mailto:sebensteg@xxxxxxx><mailto:sebensteg@xxxxxxx<mailto:sebensteg@xxxxxxx>>>
Cc: LDM
<ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx><mailto:ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx>>>;
NOAAPORT
<noaaport@xxxxxxxxxxxxxxxx<mailto:noaaport@xxxxxxxxxxxxxxxx><mailto:noaaport@xxxxxxxxxxxxxxxx<mailto:noaaport@xxxxxxxxxxxxxxxx>>>
Subject: [External] Re: [noaaport] NOAAport/SBN outage as of 13:30Z
CAUTION: This email originated from outside of COD’s system. Do not click
links, open attachments, or respond with sensitive information unless you
recognize the sender and know the content is safe.
Morning Gilbert,
I'm seeing the same and am investigating. Thanks for your report.
Best,
-Mike
Mike Zuranski
Data Engineer II
Unidata Program Center
University Corporation for Atmospheric Research
On Wed, Mar 29, 2023 at 8:43 AM Sebenste, Gilbert
<sebensteg@xxxxxxx<mailto:sebensteg@xxxxxxx><mailto:sebensteg@xxxxxxx<mailto:sebensteg@xxxxxxx>><mailto:sebensteg@xxxxxxx<mailto:sebensteg@xxxxxxx><mailto:sebensteg@xxxxxxx<mailto:sebensteg@xxxxxxx>>>>
wrote:
Getting a strong NOAAport/SBN signal here at the College of DuPage, but outside
of literally 6 products since 13:30Z, the feed has been dead.
This as of 13:42Z.
Gilbert
Gilbert Sebenste
Meteorology Support Analyst
[cid:image001.png@01D9621F.5C34C250]
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web. Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.
noaaport mailing list
noaaport@xxxxxxxxxxxxxxxx<mailto:noaaport@xxxxxxxxxxxxxxxx><mailto:noaaport@xxxxxxxxxxxxxxxx<mailto:noaaport@xxxxxxxxxxxxxxxx>><mailto:noaaport@xxxxxxxxxxxxxxxx<mailto:noaaport@xxxxxxxxxxxxxxxx><mailto:noaaport@xxxxxxxxxxxxxxxx<mailto:noaaport@xxxxxxxxxxxxxxxx>>>
For list information or to unsubscribe, visit:
https://www.unidata.ucar.edu/mailing_lists/
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web. Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.
ldm-users mailing list
ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx><mailto:ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx>>
For list information or to unsubscribe, visit:
https://www.unidata.ucar.edu/mailing_lists/
