I'm a bit mystified by your request . . . I just illustrated that the
duplicates inserted from the script were deduplicated and not sent forward,
regardless of the space or lack of space, . . . is the issue you see
downstream from there?
+----------------------------------------------------------------+
* Stonie Cooper UCAR Unidata Program *
* Software Engineer III P.O. Box 3000 *
* cooper@xxxxxxxx Boulder, CO 80307 *
* Unidata WWW Service http://www.unidata.ucar.edu/ *
+----------------------------------------------------------------+
On Wed, Mar 29, 2023 at 10:10 AM Herzmann, Daryl E [AGRON] <
akrherz@xxxxxxxxxxx> wrote:
> Stonie,
>
> Good grief, then fix LDM then to stop assuming 11 bytes to ignore and make
> it forgive the lack of a space. Why punt this issue to every IDD
> downstream user, when it can be fixed it at the source?
>
> daryl
>
> ________________________________________
> From: Stonie Cooper <cooper@xxxxxxxx>
> Sent: Wednesday, March 29, 2023 10:06 AM
> To: Herzmann, Daryl E [AGRON]
> Cc: Sebenste, Gilbert; Mike Zuranski; LDM; NOAAPORT
> Subject: Re: [ldm-users] [External] Re: [noaaport] NOAAport/SBN outage as
> of 13:30Z
>
> I'd also like to see the ICD that says there is supposed to be a space
> after the artificially applied sequence number. I can relate equal numbers
> of met data users that complain about the existence or absence of this
> particular Alden artifact, when in the end, just make your decoder
> resilient to accept either.
> +----------------------------------------------------------------+
> * Stonie Cooper UCAR Unidata Program *
> * Software Engineer III P.O. Box 3000 *
> * cooper@xxxxxxxx<mailto:cooper@xxxxxxxx>
> Boulder, CO 80307 *
> * Unidata WWW Service http://www.unidata.ucar.edu/ *
> +----------------------------------------------------------------+
>
>
> On Wed, Mar 29, 2023 at 10:00 AM Herzmann, Daryl E [AGRON] <
> akrherz@xxxxxxxxxxx<mailto:akrherz@xxxxxxxxxxx>> wrote:
> Howdy Gilbert,
>
> I'm gonna sound like a grump again, but we need to be very careful about
> putting EMWIN data onto the IDD feed. The reason is that the EMWIN and
> NOAAPort feeds are not byte-by-byte identical, so LDM has no chance to
> properly dedup these products.
>
> For example, the ADMSDM product was duplicated twice:
>
> wmo_valid_at | product_origin |
> seqnum | size
>
> ------------------------+-----------------------------------------------+----------+------
> 2023-03-29 13:54:00+00 | exo.aggregatehail.com_v_idd.agron.iastate.edu<
> http://exo.aggregatehail.com_v_idd.agron.iastate.edu> | 999 | 742
> 2023-03-29 13:54:00+00 | np2.ssec.wisc.edu_v_idd.agron.iastate.edu<
> http://np2.ssec.wisc.edu_v_idd.agron.iastate.edu> | 73462429 | 743
> 2023-03-29 14:07:00+00 | exo.aggregatehail.com_v_idd.agron.iastate.edu<
> http://exo.aggregatehail.com_v_idd.agron.iastate.edu> | 999 | 742
> 2023-03-29 14:07:00+00 | leno.unidata.ucar.edu_v_idd.agron.iastate.edu<
> http://leno.unidata.ucar.edu_v_idd.agron.iastate.edu> | 73472992 | 743
> (4 rows)
>
> As a reminder, the LDM MD5 checksum ignores the first 11 bytes of a
> product, so the "sequence number" at the top of the product does not matter
> for the MD5 calculation. The most common difference between the two feeds
> is the presence or absence of CR or LF at the end of the product, but this
> does not account for all of the differences. Another difference is the
> "Aviation Control Character" that gets stripped out.
>
> In this case, and importantly to whomever wrote the EMWIN to IDD ingest,
> is the missing space character after the sequence number:
>
> 999^M^M <-- missing space here
> NOUS42 KWNO 291407^M^M
> ADMSDM^M^M
>
> This space needs to be there for the 11 byte ignore calculation to be
> correct.
>
> It would be nice that we could turn on this level of redundancy full-time,
> but we'd likely need to condition the noaaport + EMWIN ingests to rectify
> these differences so that LDM can generate identical MD5 and properly dedup
> the feeds.
>
> daryl
>
> ________________________________________
> From: ldm-users <ldm-users-bounces@xxxxxxxxxxxxxxxx<mailto:
> ldm-users-bounces@xxxxxxxxxxxxxxxx>> on behalf of Sebenste, Gilbert <
> sebensteg@xxxxxxx<mailto:sebensteg@xxxxxxx>>
> Sent: Wednesday, March 29, 2023 9:18 AM
> To: Mike Zuranski
> Cc: LDM; NOAAPORT
> Subject: Re: [ldm-users] [External] Re: [noaaport] NOAAport/SBN outage as
> of 13:30Z
>
> AllisonHouse just turned on a kludged EMWIN feed to at least get some
> IDS|DDPLUS products going again until this outage is over with. They are
> sending it to UNIDATA, so they hope it is getting through the system.
>
> Gilbert Sebenste
> Meteorology Support Analyst
>
> [cid:image001.png@01D9621F.5C34C250]
>
> From: Mike Zuranski <mzuranski@xxxxxxxx<mailto:mzuranski@xxxxxxxx>>
> Sent: Wednesday, March 29, 2023 8:44 AM
> To: Sebenste, Gilbert <sebensteg@xxxxxxx<mailto:sebensteg@xxxxxxx>>
> Cc: LDM <ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx>>;
> NOAAPORT <noaaport@xxxxxxxxxxxxxxxx<mailto:noaaport@xxxxxxxxxxxxxxxx>>
> Subject: [External] Re: [noaaport] NOAAport/SBN outage as of 13:30Z
>
> CAUTION: This email originated from outside of COD’s system. Do not click
> links, open attachments, or respond with sensitive information unless you
> recognize the sender and know the content is safe.
>
> Morning Gilbert,
>
> I'm seeing the same and am investigating. Thanks for your report.
>
> Best,
> -Mike
>
>
> Mike Zuranski
> Data Engineer II
> Unidata Program Center
> University Corporation for Atmospheric Research
>
>
> On Wed, Mar 29, 2023 at 8:43 AM Sebenste, Gilbert <sebensteg@xxxxxxx
> <mailto:sebensteg@xxxxxxx><mailto:sebensteg@xxxxxxx<mailto:
> sebensteg@xxxxxxx>>> wrote:
> Getting a strong NOAAport/SBN signal here at the College of DuPage, but
> outside of literally 6 products since 13:30Z, the feed has been dead.
>
> This as of 13:42Z.
>
> Gilbert
>
> Gilbert Sebenste
> Meteorology Support Analyst
> [cid:image001.png@01D9621F.5C34C250]
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> noaaport mailing list
> noaaport@xxxxxxxxxxxxxxxx<mailto:noaaport@xxxxxxxxxxxxxxxx><mailto:
> noaaport@xxxxxxxxxxxxxxxx<mailto:noaaport@xxxxxxxxxxxxxxxx>>
> For list information or to unsubscribe, visit:
> https://www.unidata.ucar.edu/mailing_lists/
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> ldm-users mailing list
> ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx>
> For list information or to unsubscribe, visit:
> https://www.unidata.ucar.edu/mailing_lists/
>
