Re: [ldm-users] [External] Re: [noaaport] NOAAport/SBN outage as of 13:30Z

  • To: "Herzmann, Daryl E [AGRON]" <akrherz@xxxxxxxxxxx>
  • Subject: Re: [ldm-users] [External] Re: [noaaport] NOAAport/SBN outage as of 13:30Z
  • From: Stonie Cooper <cooper@xxxxxxxx>
  • Date: Wed, 29 Mar 2023 10:16:14 -0500
I'm a bit mystified by your request . . . I just illustrated that the
duplicates inserted from the script were deduplicated and not sent forward,
regardless of the space or lack of space, . . . is the issue you see
downstream from there?
+----------------------------------------------------------------+
* Stonie Cooper                             UCAR Unidata Program *
* Software Engineer III                            P.O. Box 3000 *
* cooper@xxxxxxxx                              Boulder, CO 80307 *
* Unidata WWW Service               http://www.unidata.ucar.edu/ *
+----------------------------------------------------------------+


On Wed, Mar 29, 2023 at 10:10 AM Herzmann, Daryl E [AGRON] <
akrherz@xxxxxxxxxxx> wrote:

> Stonie,
>
> Good grief, then fix LDM then to stop assuming 11 bytes to ignore and make
> it forgive the lack of a space.  Why punt this issue to every IDD
> downstream user, when it can be fixed it at the source?
>
> daryl
>
> ________________________________________
> From: Stonie Cooper <cooper@xxxxxxxx>
> Sent: Wednesday, March 29, 2023 10:06 AM
> To: Herzmann, Daryl E [AGRON]
> Cc: Sebenste, Gilbert; Mike Zuranski; LDM; NOAAPORT
> Subject: Re: [ldm-users] [External] Re: [noaaport] NOAAport/SBN outage as
> of 13:30Z
>
> I'd also like to see the ICD that says there is supposed to be a space
> after the artificially applied sequence number.  I can relate equal numbers
> of met data users that complain about the existence or absence of this
> particular Alden artifact, when in the end, just make your decoder
> resilient to accept either.
> +----------------------------------------------------------------+
> * Stonie Cooper                             UCAR Unidata Program *
> * Software Engineer III                            P.O. Box 3000 *
> * cooper@xxxxxxxx<mailto:cooper@xxxxxxxx>
> Boulder, CO 80307 *
> * Unidata WWW Service               http://www.unidata.ucar.edu/ *
> +----------------------------------------------------------------+
>
>
> On Wed, Mar 29, 2023 at 10:00 AM Herzmann, Daryl E [AGRON] <
> akrherz@xxxxxxxxxxx<mailto:akrherz@xxxxxxxxxxx>> wrote:
> Howdy Gilbert,
>
> I'm gonna sound like a grump again, but we need to be very careful about
> putting EMWIN data onto the IDD feed.  The reason is that the EMWIN and
> NOAAPort feeds are not byte-by-byte identical, so LDM has no chance to
> properly dedup these products.
>
> For example, the ADMSDM product was duplicated twice:
>
>       wmo_valid_at      |                product_origin                 |
> seqnum  | size
>
> ------------------------+-----------------------------------------------+----------+------
>  2023-03-29 13:54:00+00 | exo.aggregatehail.com_v_idd.agron.iastate.edu<
> http://exo.aggregatehail.com_v_idd.agron.iastate.edu> |      999 |  742
>  2023-03-29 13:54:00+00 | np2.ssec.wisc.edu_v_idd.agron.iastate.edu<
> http://np2.ssec.wisc.edu_v_idd.agron.iastate.edu>     | 73462429 |  743
>  2023-03-29 14:07:00+00 | exo.aggregatehail.com_v_idd.agron.iastate.edu<
> http://exo.aggregatehail.com_v_idd.agron.iastate.edu> |      999 |  742
>  2023-03-29 14:07:00+00 | leno.unidata.ucar.edu_v_idd.agron.iastate.edu<
> http://leno.unidata.ucar.edu_v_idd.agron.iastate.edu> | 73472992 |  743
> (4 rows)
>
> As a reminder, the LDM MD5 checksum ignores the first 11 bytes of a
> product, so the "sequence number" at the top of the product does not matter
> for the MD5 calculation.  The most common difference between the two feeds
> is the presence or absence of CR or LF at the end of the product, but this
> does not account for all of the differences.  Another difference is the
> "Aviation Control Character" that gets stripped out.
>
> In this case, and importantly to whomever wrote the EMWIN to IDD ingest,
> is the missing space character after the sequence number:
>
> 999^M^M  <-- missing space here
> NOUS42 KWNO 291407^M^M
> ADMSDM^M^M
>
> This space needs to be there for the 11 byte ignore calculation to be
> correct.
>
> It would be nice that we could turn on this level of redundancy full-time,
> but we'd likely need to condition the noaaport + EMWIN ingests to rectify
> these differences so that LDM can generate identical MD5 and properly dedup
> the feeds.
>
> daryl
>
> ________________________________________
> From: ldm-users <ldm-users-bounces@xxxxxxxxxxxxxxxx<mailto:
> ldm-users-bounces@xxxxxxxxxxxxxxxx>> on behalf of Sebenste, Gilbert <
> sebensteg@xxxxxxx<mailto:sebensteg@xxxxxxx>>
> Sent: Wednesday, March 29, 2023 9:18 AM
> To: Mike Zuranski
> Cc: LDM; NOAAPORT
> Subject: Re: [ldm-users] [External] Re: [noaaport] NOAAport/SBN outage as
> of 13:30Z
>
> AllisonHouse just turned on a kludged EMWIN feed to at least get some
> IDS|DDPLUS products going again until this outage is over with. They are
> sending it to UNIDATA, so they hope it is getting through the system.
>
> Gilbert Sebenste
> Meteorology Support Analyst
>
> [cid:image001.png@01D9621F.5C34C250]
>
> From: Mike Zuranski <mzuranski@xxxxxxxx<mailto:mzuranski@xxxxxxxx>>
> Sent: Wednesday, March 29, 2023 8:44 AM
> To: Sebenste, Gilbert <sebensteg@xxxxxxx<mailto:sebensteg@xxxxxxx>>
> Cc: LDM <ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx>>;
> NOAAPORT <noaaport@xxxxxxxxxxxxxxxx<mailto:noaaport@xxxxxxxxxxxxxxxx>>
> Subject: [External] Re: [noaaport] NOAAport/SBN outage as of 13:30Z
>
> CAUTION: This email originated from outside of COD’s system. Do not click
> links, open attachments, or respond with sensitive information unless you
> recognize the sender and know the content is safe.
>
> Morning Gilbert,
>
> I'm seeing the same and am investigating.  Thanks for your report.
>
> Best,
> -Mike
>
>
> Mike Zuranski
> Data Engineer II
> Unidata Program Center
> University Corporation for Atmospheric Research
>
>
> On Wed, Mar 29, 2023 at 8:43 AM Sebenste, Gilbert <sebensteg@xxxxxxx
> <mailto:sebensteg@xxxxxxx><mailto:sebensteg@xxxxxxx<mailto:
> sebensteg@xxxxxxx>>> wrote:
> Getting a strong NOAAport/SBN signal here at the College of DuPage, but
> outside of literally 6 products since 13:30Z, the feed has been dead.
>
> This as of 13:42Z.
>
> Gilbert
>
> Gilbert Sebenste
> Meteorology Support Analyst
> [cid:image001.png@01D9621F.5C34C250]
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web.  Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> noaaport mailing list
> noaaport@xxxxxxxxxxxxxxxx<mailto:noaaport@xxxxxxxxxxxxxxxx><mailto:
> noaaport@xxxxxxxxxxxxxxxx<mailto:noaaport@xxxxxxxxxxxxxxxx>>
> For list information or to unsubscribe, visit:
> https://www.unidata.ucar.edu/mailing_lists/
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web.  Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> ldm-users mailing list
> ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx>
> For list information or to unsubscribe,  visit:
> https://www.unidata.ucar.edu/mailing_lists/
>
  • 2023 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the ldm-users archives: