We have a socket feed from the TOC at AWC. We request about a third of the
text products that are in the WMO feed for our socket feed as a backup for
when NOAAPort goes down. The socket feed is missing that space after the
sequence numbers. So when we moved from using an old AWIPS ingestor to the
LDM NOAAPort ingester, we got a lot of duplicates because of that added
space. We also noted an extra CR/CR/LF on the text products. That made the
text products from LDM NOAAPort 4 bytes longer than we were seeing from the
socket feed from the TOC. Since we didn't control the software that
ingested the TOC socket feed, I ended up modifying the productMaker code in
the LDM. I removed the space after the sequence number and added code to
not add a CR/CR/LF if the product already ends with that sequence. As a
result, I don't do this for binary products (BUFR, GRIB) that don't have a
trailing CR/CR/LF in the raw NOAAPort stream and need those added. That got
rid of our duplicate problem.
I did contribute my changes back to Unidata sometime in the 2013/14
timeframe but they considered it a one site modification and didn't apply
it to the LDM code base. So I now do that change for each new LDM version
we deploy on our NOAAPort receive LDM. It's only a two line change. So it's
easy for me to do it.
Dan.
On Wed, Mar 29, 2023 at 10:23 AM Sebenste, Gilbert <sebensteg@xxxxxxx>
wrote:
> Interesting. LDM 6.14.5 rejected the duplicates here. (shrug)
>
>
>
> Gilbert Sebenste
>
> Meteorology Support Analyst
>
>
>
> *From:* Stonie Cooper <cooper@xxxxxxxx>
> *Sent:* Wednesday, March 29, 2023 10:20 AM
> *To:* Sebenste, Gilbert <sebensteg@xxxxxxx>
> *Cc:* Herzmann, Daryl E [AGRON] <akrherz@xxxxxxxxxxx>; Mike Zuranski <
> mzuranski@xxxxxxxx>; LDM <ldm-users@xxxxxxxxxxxxxxxx>; NOAAPORT <
> noaaport@xxxxxxxxxxxxxxxx>
> *Subject:* Re: [ldm-users] [External] Re: [noaaport] NOAAport/SBN outage
> as of 13:30Z
>
>
>
> Daryl indicated that they were not deduplicated in his setup . . . which
> requires additional investigation. . . .
>
> +----------------------------------------------------------------+
> * Stonie Cooper UCAR Unidata Program *
> * Software Engineer III P.O. Box 3000 *
> * cooper@xxxxxxxx Boulder, CO 80307 *
> * Unidata WWW Service http://www.unidata.ucar.edu/ *
> +----------------------------------------------------------------+
>
>
>
>
>
> On Wed, Mar 29, 2023 at 10:18 AM Sebenste, Gilbert <sebensteg@xxxxxxx>
> wrote:
>
> Because that shouldn't be the LDM's issue. If the NWS is inconsistent
> sending out products, that should be on them to fix, not Stonie or anyone
> else.
> In either case, what is the issue? Again, Stonie showed that the
> duplication of products were stopped...so the LDM is, in fact, doing what
> you want.
>
> Gilbert
>
> Gilbert Sebenste
> Meteorology Support Analyst
>
>
>
> -----Original Message-----
> From: Herzmann, Daryl E [AGRON] <akrherz@xxxxxxxxxxx>
> Sent: Wednesday, March 29, 2023 10:11 AM
> To: Stonie Cooper <cooper@xxxxxxxx>
> Cc: Sebenste, Gilbert <sebensteg@xxxxxxx>; Mike Zuranski <
> mzuranski@xxxxxxxx>; LDM <ldm-users@xxxxxxxxxxxxxxxx>; NOAAPORT <
> noaaport@xxxxxxxxxxxxxxxx>
> Subject: Re: [ldm-users] [External] Re: [noaaport] NOAAport/SBN outage as
> of 13:30Z
>
> Stonie,
>
> Good grief, then fix LDM then to stop assuming 11 bytes to ignore and make
> it forgive the lack of a space. Why punt this issue to every IDD
> downstream user, when it can be fixed it at the source?
>
> daryl
>
> ________________________________________
> From: Stonie Cooper <cooper@xxxxxxxx>
> Sent: Wednesday, March 29, 2023 10:06 AM
> To: Herzmann, Daryl E [AGRON]
> Cc: Sebenste, Gilbert; Mike Zuranski; LDM; NOAAPORT
> Subject: Re: [ldm-users] [External] Re: [noaaport] NOAAport/SBN outage as
> of 13:30Z
>
> I'd also like to see the ICD that says there is supposed to be a space
> after the artificially applied sequence number. I can relate equal numbers
> of met data users that complain about the existence or absence of this
> particular Alden artifact, when in the end, just make your decoder
> resilient to accept either.
> +----------------------------------------------------------------+
> * Stonie Cooper UCAR Unidata Program *
> * Software Engineer III P.O. Box 3000 *
> * cooper@xxxxxxxx<mailto:cooper@xxxxxxxx>
> Boulder, CO 80307 *
> * Unidata WWW Service http://www.unidata.ucar.edu/ *
> +----------------------------------------------------------------+
>
>
> On Wed, Mar 29, 2023 at 10:00 AM Herzmann, Daryl E [AGRON] <
> akrherz@xxxxxxxxxxx<mailto:akrherz@xxxxxxxxxxx>> wrote:
> Howdy Gilbert,
>
> I'm gonna sound like a grump again, but we need to be very careful about
> putting EMWIN data onto the IDD feed. The reason is that the EMWIN and
> NOAAPort feeds are not byte-by-byte identical, so LDM has no chance to
> properly dedup these products.
>
> For example, the ADMSDM product was duplicated twice:
>
> wmo_valid_at | product_origin |
> seqnum | size
>
> ------------------------+-----------------------------------------------+----------+------
> 2023-03-29 13:54:00+00 | exo.aggregatehail.com_v_idd.agron.iastate.edu<
> http://exo.aggregatehail.com_v_idd.agron.iastate.edu> | 999 | 742
> 2023-03-29 13:54:00+00 | np2.ssec.wisc.edu_v_idd.agron.iastate.edu<
> http://np2.ssec.wisc.edu_v_idd.agron.iastate.edu> | 73462429 | 743
> 2023-03-29 14:07:00+00 | exo.aggregatehail.com_v_idd.agron.iastate.edu<
> http://exo.aggregatehail.com_v_idd.agron.iastate.edu> | 999 | 742
> 2023-03-29 14:07:00+00 | leno.unidata.ucar.edu_v_idd.agron.iastate.edu<
> http://leno.unidata.ucar.edu_v_idd.agron.iastate.edu> | 73472992 | 743
> (4 rows)
>
> As a reminder, the LDM MD5 checksum ignores the first 11 bytes of a
> product, so the "sequence number" at the top of the product does not matter
> for the MD5 calculation. The most common difference between the two feeds
> is the presence or absence of CR or LF at the end of the product, but this
> does not account for all of the differences. Another difference is the
> "Aviation Control Character" that gets stripped out.
>
> In this case, and importantly to whomever wrote the EMWIN to IDD ingest,
> is the missing space character after the sequence number:
>
> 999^M^M <-- missing space here
> NOUS42 KWNO 291407^M^M
> ADMSDM^M^M
>
> This space needs to be there for the 11 byte ignore calculation to be
> correct.
>
> It would be nice that we could turn on this level of redundancy full-time,
> but we'd likely need to condition the noaaport + EMWIN ingests to rectify
> these differences so that LDM can generate identical MD5 and properly dedup
> the feeds.
>
> daryl
>
> ________________________________________
> From: ldm-users <ldm-users-bounces@xxxxxxxxxxxxxxxx<mailto:
> ldm-users-bounces@xxxxxxxxxxxxxxxx>> on behalf of Sebenste, Gilbert <
> sebensteg@xxxxxxx<mailto:sebensteg@xxxxxxx>>
> Sent: Wednesday, March 29, 2023 9:18 AM
> To: Mike Zuranski
> Cc: LDM; NOAAPORT
> Subject: Re: [ldm-users] [External] Re: [noaaport] NOAAport/SBN outage as
> of 13:30Z
>
> AllisonHouse just turned on a kludged EMWIN feed to at least get some
> IDS|DDPLUS products going again until this outage is over with. They are
> sending it to UNIDATA, so they hope it is getting through the system.
>
> Gilbert Sebenste
> Meteorology Support Analyst
>
> [cid:image001.png@01D9621F.5C34C250]
>
> From: Mike Zuranski <mzuranski@xxxxxxxx<mailto:mzuranski@xxxxxxxx>>
> Sent: Wednesday, March 29, 2023 8:44 AM
> To: Sebenste, Gilbert <sebensteg@xxxxxxx<mailto:sebensteg@xxxxxxx>>
> Cc: LDM <ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx>>;
> NOAAPORT <noaaport@xxxxxxxxxxxxxxxx<mailto:noaaport@xxxxxxxxxxxxxxxx>>
> Subject: [External] Re: [noaaport] NOAAport/SBN outage as of 13:30Z
>
> CAUTION: This email originated from outside of COD’s system. Do not click
> links, open attachments, or respond with sensitive information unless you
> recognize the sender and know the content is safe.
>
> Morning Gilbert,
>
> I'm seeing the same and am investigating. Thanks for your report.
>
> Best,
> -Mike
>
>
> Mike Zuranski
> Data Engineer II
> Unidata Program Center
> University Corporation for Atmospheric Research
>
>
> On Wed, Mar 29, 2023 at 8:43 AM Sebenste, Gilbert <sebensteg@xxxxxxx
> <mailto:sebensteg@xxxxxxx><mailto:sebensteg@xxxxxxx<mailto:
> sebensteg@xxxxxxx>>> wrote:
> Getting a strong NOAAport/SBN signal here at the College of DuPage, but
> outside of literally 6 products since 13:30Z, the feed has been dead.
>
> This as of 13:42Z.
>
> Gilbert
>
> Gilbert Sebenste
> Meteorology Support Analyst
> [cid:image001.png@01D9621F.5C34C250]
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are recorded
> in the Unidata inquiry tracking system and made publicly available through
> the web. Users who post to any of the lists we maintain are reminded to
> remove any personal information that they do not want to be made public.
>
>
> noaaport mailing list
> noaaport@xxxxxxxxxxxxxxxx<mailto:noaaport@xxxxxxxxxxxxxxxx><mailto:
> noaaport@xxxxxxxxxxxxxxxx<mailto:noaaport@xxxxxxxxxxxxxxxx>>
> For list information or to unsubscribe, visit:
> https://www.unidata.ucar.edu/mailing_lists/
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are recorded
> in the Unidata inquiry tracking system and made publicly available through
> the web. Users who post to any of the lists we maintain are reminded to
> remove any personal information that they do not want to be made public.
>
>
> ldm-users mailing list
> ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx>
> For list information or to unsubscribe, visit:
> https://www.unidata.ucar.edu/mailing_lists/
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> ldm-users mailing list
> ldm-users@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> https://www.unidata.ucar.edu/mailing_lists/
>
--
*Dan Vietor*
*Senior Research Meteorologist*
CIRA, Colorado State Univ
Aviation Weather Center
Kansas City, MO
816.584.7211
