Re: [ldm-users] [External] Re: [noaaport] NOAAport/SBN outage as of 13:30Z

  • To: Daniel Vietor - NOAA Affiliate <dan.vietor@xxxxxxxx>
  • Subject: Re: [ldm-users] [External] Re: [noaaport] NOAAport/SBN outage as of 13:30Z
  • From: "Sebenste, Gilbert" <sebensteg@xxxxxxx>
  • Date: Wed, 29 Mar 2023 15:53:55 +0000
Ahhhh, I wonder if the EMWIN feed is made by the socket feed from the TOC? That 
would explain all of this, and why it was the only feed that also stayed up and 
running.

Gilbert

Gilbert Sebenste
Meteorology Support Analyst

[cid:image001.png@01D9622C.C153FBD0]

From: Daniel Vietor - NOAA Affiliate <dan.vietor@xxxxxxxx>
Sent: Wednesday, March 29, 2023 10:51 AM
To: Sebenste, Gilbert <sebensteg@xxxxxxx>
Cc: Stonie Cooper <cooper@xxxxxxxx>; NOAAPORT <noaaport@xxxxxxxxxxxxxxxx>; LDM 
<ldm-users@xxxxxxxxxxxxxxxx>
Subject: Re: [ldm-users] [External] Re: [noaaport] NOAAport/SBN outage as of 
13:30Z

We have a socket feed from the TOC at AWC. We request about a third of the text 
products that are in the WMO feed for our socket feed as a backup for when 
NOAAPort goes down. The socket feed is missing that space after the sequence 
numbers. So when we moved from using an old AWIPS ingestor to the LDM NOAAPort 
ingester, we got a lot of duplicates because of that added space. We also noted 
an extra CR/CR/LF on the text products. That made the text products from LDM 
NOAAPort 4 bytes longer than we were seeing from the socket feed from the TOC. 
Since we didn't control the software that ingested the TOC socket feed, I ended 
up modifying the productMaker code in the LDM. I removed the space after the 
sequence number and added code to not add a CR/CR/LF if the product already 
ends with that sequence. As a result, I don't do this for binary products 
(BUFR, GRIB) that don't have a trailing CR/CR/LF in the raw NOAAPort stream and 
need those added. That got rid of our duplicate problem.
I did contribute my changes back to Unidata sometime in the 2013/14 timeframe 
but they considered it a one site modification and didn't apply it to the LDM 
code base. So I now do that change for each new LDM version we deploy on our 
NOAAPort receive LDM. It's only a two line change. So it's easy for me to do it.

Dan.

On Wed, Mar 29, 2023 at 10:23 AM Sebenste, Gilbert 
<sebensteg@xxxxxxx<mailto:sebensteg@xxxxxxx>> wrote:
Interesting. LDM 6.14.5 rejected the duplicates here. (shrug)

Gilbert Sebenste
Meteorology Support Analyst
[cid:image001.png@01D9622C.C153FBD0]

From: Stonie Cooper <cooper@xxxxxxxx<mailto:cooper@xxxxxxxx>>
Sent: Wednesday, March 29, 2023 10:20 AM
To: Sebenste, Gilbert <sebensteg@xxxxxxx<mailto:sebensteg@xxxxxxx>>
Cc: Herzmann, Daryl E [AGRON] 
<akrherz@xxxxxxxxxxx<mailto:akrherz@xxxxxxxxxxx>>; Mike Zuranski 
<mzuranski@xxxxxxxx<mailto:mzuranski@xxxxxxxx>>; LDM 
<ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx>>; NOAAPORT 
<noaaport@xxxxxxxxxxxxxxxx<mailto:noaaport@xxxxxxxxxxxxxxxx>>
Subject: Re: [ldm-users] [External] Re: [noaaport] NOAAport/SBN outage as of 
13:30Z

Daryl indicated that they were not deduplicated in his setup . . . which 
requires additional investigation. . . .
+----------------------------------------------------------------+
* Stonie Cooper                             UCAR Unidata Program *
* Software Engineer III                            P.O. Box 3000 *
* cooper@xxxxxxxx<mailto:cooper@xxxxxxxx>                              Boulder, 
CO 80307 *
* Unidata WWW Service               http://www.unidata.ucar.edu/ *
+----------------------------------------------------------------+


On Wed, Mar 29, 2023 at 10:18 AM Sebenste, Gilbert 
<sebensteg@xxxxxxx<mailto:sebensteg@xxxxxxx>> wrote:
Because that shouldn't be the LDM's issue. If the NWS is inconsistent sending 
out products, that should be on them to fix, not Stonie or anyone else.
In either case, what is the issue? Again, Stonie showed that the duplication of 
products were stopped...so the LDM is, in fact, doing what you want.

Gilbert

Gilbert Sebenste
Meteorology Support Analyst



-----Original Message-----
From: Herzmann, Daryl E [AGRON] 
<akrherz@xxxxxxxxxxx<mailto:akrherz@xxxxxxxxxxx>>
Sent: Wednesday, March 29, 2023 10:11 AM
To: Stonie Cooper <cooper@xxxxxxxx<mailto:cooper@xxxxxxxx>>
Cc: Sebenste, Gilbert <sebensteg@xxxxxxx<mailto:sebensteg@xxxxxxx>>; Mike 
Zuranski <mzuranski@xxxxxxxx<mailto:mzuranski@xxxxxxxx>>; LDM 
<ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx>>; NOAAPORT 
<noaaport@xxxxxxxxxxxxxxxx<mailto:noaaport@xxxxxxxxxxxxxxxx>>
Subject: Re: [ldm-users] [External] Re: [noaaport] NOAAport/SBN outage as of 
13:30Z

Stonie,

Good grief, then fix LDM then to stop assuming 11 bytes to ignore and make it 
forgive the lack of a space.  Why punt this issue to every IDD downstream user, 
when it can be fixed it at the source?

daryl

________________________________________
From: Stonie Cooper <cooper@xxxxxxxx<mailto:cooper@xxxxxxxx>>
Sent: Wednesday, March 29, 2023 10:06 AM
To: Herzmann, Daryl E [AGRON]
Cc: Sebenste, Gilbert; Mike Zuranski; LDM; NOAAPORT
Subject: Re: [ldm-users] [External] Re: [noaaport] NOAAport/SBN outage as of 
13:30Z

I'd also like to see the ICD that says there is supposed to be a space after 
the artificially applied sequence number.  I can relate equal numbers of met 
data users that complain about the existence or absence of this particular 
Alden artifact, when in the end, just make your decoder resilient to accept 
either.
+----------------------------------------------------------------+
* Stonie Cooper                             UCAR Unidata Program *
* Software Engineer III                            P.O. Box 3000 *
* 
cooper@xxxxxxxx<mailto:cooper@xxxxxxxx><mailto:cooper@xxxxxxxx<mailto:cooper@xxxxxxxx>>
                              Boulder, CO 80307 *
* Unidata WWW Service               http://www.unidata.ucar.edu/ *
+----------------------------------------------------------------+


On Wed, Mar 29, 2023 at 10:00 AM Herzmann, Daryl E [AGRON] 
<akrherz@xxxxxxxxxxx<mailto:akrherz@xxxxxxxxxxx><mailto:akrherz@xxxxxxxxxxx<mailto:akrherz@xxxxxxxxxxx>>>
 wrote:
Howdy Gilbert,

I'm gonna sound like a grump again, but we need to be very careful about 
putting EMWIN data onto the IDD feed.  The reason is that the EMWIN and 
NOAAPort feeds are not byte-by-byte identical, so LDM has no chance to properly 
dedup these products.

For example, the ADMSDM product was duplicated twice:

      wmo_valid_at      |                product_origin                 |  
seqnum  | size
------------------------+-----------------------------------------------+----------+------
 2023-03-29 13:54:00+00 | 
exo.aggregatehail.com_v_idd.agron.iastate.edu<http://exo.aggregatehail.com_v_idd.agron.iastate.edu><http://exo.aggregatehail.com_v_idd.agron.iastate.edu>
 |      999 |  742
 2023-03-29 13:54:00+00 | 
np2.ssec.wisc.edu_v_idd.agron.iastate.edu<http://np2.ssec.wisc.edu_v_idd.agron.iastate.edu><http://np2.ssec.wisc.edu_v_idd.agron.iastate.edu>
     | 73462429 |  743
 2023-03-29 14:07:00+00 | 
exo.aggregatehail.com_v_idd.agron.iastate.edu<http://exo.aggregatehail.com_v_idd.agron.iastate.edu><http://exo.aggregatehail.com_v_idd.agron.iastate.edu>
 |      999 |  742
 2023-03-29 14:07:00+00 | 
leno.unidata.ucar.edu_v_idd.agron.iastate.edu<http://leno.unidata.ucar.edu_v_idd.agron.iastate.edu><http://leno.unidata.ucar.edu_v_idd.agron.iastate.edu>
 | 73472992 |  743
(4 rows)

As a reminder, the LDM MD5 checksum ignores the first 11 bytes of a product, so 
the "sequence number" at the top of the product does not matter for the MD5 
calculation.  The most common difference between the two feeds is the presence 
or absence of CR or LF at the end of the product, but this does not account for 
all of the differences.  Another difference is the "Aviation Control Character" 
that gets stripped out.

In this case, and importantly to whomever wrote the EMWIN to IDD ingest, is the 
missing space character after the sequence number:

999^M^M  <-- missing space here
NOUS42 KWNO 291407^M^M
ADMSDM^M^M

This space needs to be there for the 11 byte ignore calculation to be correct.

It would be nice that we could turn on this level of redundancy full-time, but 
we'd likely need to condition the noaaport + EMWIN ingests to rectify these 
differences so that LDM can generate identical MD5 and properly dedup the feeds.

daryl

________________________________________
From: ldm-users 
<ldm-users-bounces@xxxxxxxxxxxxxxxx<mailto:ldm-users-bounces@xxxxxxxxxxxxxxxx><mailto:ldm-users-bounces@xxxxxxxxxxxxxxxx<mailto:ldm-users-bounces@xxxxxxxxxxxxxxxx>>>
 on behalf of Sebenste, Gilbert 
<sebensteg@xxxxxxx<mailto:sebensteg@xxxxxxx><mailto:sebensteg@xxxxxxx<mailto:sebensteg@xxxxxxx>>>
Sent: Wednesday, March 29, 2023 9:18 AM
To: Mike Zuranski
Cc: LDM; NOAAPORT
Subject: Re: [ldm-users] [External] Re: [noaaport] NOAAport/SBN outage as of 
13:30Z

AllisonHouse just turned on a kludged EMWIN feed to at least get some 
IDS|DDPLUS products going again until this outage is over with. They are 
sending it to UNIDATA, so they hope it is getting through the system.

Gilbert Sebenste
Meteorology Support Analyst

[cid:image001.png@01D9621F.5C34C250]

From: Mike Zuranski 
<mzuranski@xxxxxxxx<mailto:mzuranski@xxxxxxxx><mailto:mzuranski@xxxxxxxx<mailto:mzuranski@xxxxxxxx>>>
Sent: Wednesday, March 29, 2023 8:44 AM
To: Sebenste, Gilbert 
<sebensteg@xxxxxxx<mailto:sebensteg@xxxxxxx><mailto:sebensteg@xxxxxxx<mailto:sebensteg@xxxxxxx>>>
Cc: LDM 
<ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx><mailto:ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx>>>;
 NOAAPORT 
<noaaport@xxxxxxxxxxxxxxxx<mailto:noaaport@xxxxxxxxxxxxxxxx><mailto:noaaport@xxxxxxxxxxxxxxxx<mailto:noaaport@xxxxxxxxxxxxxxxx>>>
Subject: [External] Re: [noaaport] NOAAport/SBN outage as of 13:30Z

CAUTION: This email originated from outside of COD’s system. Do not click 
links, open attachments, or respond with sensitive information unless you 
recognize the sender and know the content is safe.

Morning Gilbert,

I'm seeing the same and am investigating.  Thanks for your report.

Best,
-Mike


Mike Zuranski
Data Engineer II
Unidata Program Center
University Corporation for Atmospheric Research


On Wed, Mar 29, 2023 at 8:43 AM Sebenste, Gilbert 
<sebensteg@xxxxxxx<mailto:sebensteg@xxxxxxx><mailto:sebensteg@xxxxxxx<mailto:sebensteg@xxxxxxx>><mailto:sebensteg@xxxxxxx<mailto:sebensteg@xxxxxxx><mailto:sebensteg@xxxxxxx<mailto:sebensteg@xxxxxxx>>>>
 wrote:
Getting a strong NOAAport/SBN signal here at the College of DuPage, but outside 
of literally 6 products since 13:30Z, the feed has been dead.

This as of 13:42Z.

Gilbert

Gilbert Sebenste
Meteorology Support Analyst
[cid:image001.png@01D9621F.5C34C250]

_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are recorded in 
the Unidata inquiry tracking system and made publicly available through the 
web.  Users who post to any of the lists we maintain are reminded to remove any 
personal information that they do not want to be made public.


noaaport mailing list
noaaport@xxxxxxxxxxxxxxxx<mailto:noaaport@xxxxxxxxxxxxxxxx><mailto:noaaport@xxxxxxxxxxxxxxxx<mailto:noaaport@xxxxxxxxxxxxxxxx>><mailto:noaaport@xxxxxxxxxxxxxxxx<mailto:noaaport@xxxxxxxxxxxxxxxx><mailto:noaaport@xxxxxxxxxxxxxxxx<mailto:noaaport@xxxxxxxxxxxxxxxx>>>
For list information or to unsubscribe, visit: 
https://www.unidata.ucar.edu/mailing_lists/
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are recorded in 
the Unidata inquiry tracking system and made publicly available through the 
web.  Users who post to any of the lists we maintain are reminded to remove any 
personal information that they do not want to be made public.


ldm-users mailing list
ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx><mailto:ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx>>
For list information or to unsubscribe,  visit: 
https://www.unidata.ucar.edu/mailing_lists/
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web.  Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.


ldm-users mailing list
ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx>
For list information or to unsubscribe,  visit: 
https://www.unidata.ucar.edu/mailing_lists/


--
Dan Vietor
Senior Research Meteorologist
CIRA, Colorado State Univ
Aviation Weather Center
Kansas City, MO
816.584.7211

PNG image

  • 2023 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the ldm-users archives: