Re: [ldm-users] [External] Re: [noaaport] NOAAport/SBN outage as of 13:30Z

  • To: Stonie Cooper <cooper@xxxxxxxx>
  • Subject: Re: [ldm-users] [External] Re: [noaaport] NOAAport/SBN outage as of 13:30Z
  • From: "Herzmann, Daryl E [AGRON]" <akrherz@xxxxxxxxxxx>
  • Date: Wed, 29 Mar 2023 15:04:11 +0000
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=iastate.edu; dmarc=pass action=none header.from=iastate.edu; dkim=pass header.d=iastate.edu; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=dSWxxFxBAmO4i+4hTKpzSikgXfJ6MIfqmmFtu69VlEs=; b=aQDny5STX8R8SEhIBRCrwuLzjf39gM/p2Hu9oVgzxupUl20olnKdV8y9dOCGSGNuvVWmLoHl5NhX6g5upn1w44URO86P+o6hP1lI/RRT0T8hGk5IdNfDig3czr7GbrKod2DvbYR10iTuQnxGpYthFC9ocDi1h+pWLrFiMw34ewWN3bW9CBLdTz9ddY4/HGM0YqPksoSi0YqJ1k6uXPKysEYmcujAb2DGnEOgVANG5b0nJfPDpLdr51JsbRJr2RlslD8tsGG14z1v/uRZnOlp3V5RBh53PjPvJu7pGKhf5YInfpCBKCvpQ84OofIzl+TW2HFlPAVNyrRM8mVlNOxKRg==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=RCO4P/Y+IfkWbHaljeAFgOSdH0jrqWsWaOhwlZuPrmnooiWP8gqzgty6YCDD5l4VJz8rJ55VjWPIWytYOOV6IocwRdUTOeIu/sdKmzcEh524iSo3CXMOI7ImYeNh/YuOH+n2M79w89OT2ixO8KGKvS3G7ubu7YT8680qkbRNK7WNuMlaFVnhvhGAiwyi822kuh90iJGDy9F5NV5A/v+y9wivCiwLMCf6C8kJhxAErlAIvBPY2utxbiwcNDj2TB/fBGxVk3Nw1dD2cgisZVE2j/5/xqHamss6/IYL+jlfM5tNBFjQb/O2QEQeT/bPoyr4KDSwLKhSLw1dxRa467S5RQ==
  • Authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=iastate.edu;
  • Msip_labels:
Howdy Stonie,

You disagree there should be a space after the 999 ldm sequence number to get 
the LDM 11 byte MD5 checksum correct?

daryl

________________________________________
From: Stonie Cooper <cooper@xxxxxxxx>
Sent: Wednesday, March 29, 2023 10:01 AM
To: Herzmann, Daryl E [AGRON]
Cc: Sebenste, Gilbert; Mike Zuranski; LDM; NOAAPORT
Subject: Re: [ldm-users] [External] Re: [noaaport] NOAAport/SBN outage as of 
13:30Z

I disagree.  And to illustrate, this is what happens when you run the emwin 
insertion:

20230329T145036.073685Z pqinsert[2687]              pqinsert.c:main:443         
        ERROR Product already in queue: 1dc1a8be4c16707deb53b8b54c91e9d2        
 89 20230329145036.073608 DDPLUS 000  SAUS80 KWBC 291500 /pMETAR
20230329T145036.240065Z pqinsert[2939]              pqinsert.c:main:443         
        ERROR Product already in queue: 4ec193533d40792d288ad39669779d86       
1299 20230329145036.239987 DDPLUS 000  FPUS63 KBIS 291447 /pSFPND
20230329T145036.481866Z pqinsert[3302]              pqinsert.c:main:443         
        ERROR Product already in queue: 6466efe262a46bd5941ab7fdf92b77ab        
963 20230329145036.481785 DDPLUS 000  SPUS70 KWBC 291446 /pSPECI
20230329T145036.642540Z pqinsert[3545]              pqinsert.c:main:443         
        ERROR Product already in queue: b630c518ccbcd090b5dbb853eeec232c        
242 20230329145036.642460 DDPLUS 000  SAUS80 KWBC 291446 /pMETAR
20230329T145036.727266Z pqinsert[3671]              pqinsert.c:main:443         
        ERROR Product already in queue: ffbbda56d34ae475e4da727428f76c98        
413 20230329145036.727189 DDPLUS 000  SRUS43 KKRF 291446 /pRVMKRF
20230329T145037.245965Z pqinsert[4474]              pqinsert.c:main:443         
        ERROR Product already in queue: 3839d746854a538f956433100b581f20        
162 20230329145037.245882 DDPLUS 000  NOUS64 KMOB 291447 /pFTMMOB
20230329T145037.769953Z pqinsert[5261]              pqinsert.c:main:443         
        ERROR Product already in queue: 6e418199e50d54393474f1bbf6d0d8e0       
2008 20230329145037.769858 DDPLUS 000  FLUS43 KBIS 291446 /pHWOBIS
20230329T145037.890334Z pqinsert[5443]              pqinsert.c:main:443         
        ERROR Product already in queue: 0486d0d6331694e523d4ff36d6d6f2db       
3373 20230329145037.890259 DDPLUS 000  ASUS65 KABQ 291447 /pRTPNM
20230329T145038.180060Z pqinsert[5870]              pqinsert.c:main:443         
        ERROR Product already in queue: 8e183a08bdf5c3f51937787a20521313       
2514 20230329145038.179980 DDPLUS 000  SAUS70 KWBC 291446 /pMETAR

The deduplication nature of the LDM will filter out the second copy.  In the 
end, having a more resilient decoding apparatus is more important than possibly 
getting duplicates as compared to none.

Stonie
+----------------------------------------------------------------+
* Stonie Cooper                             UCAR Unidata Program *
* Software Engineer III                            P.O. Box 3000 *
* cooper@xxxxxxxx<mailto:cooper@xxxxxxxx>                              Boulder, 
CO 80307 *
* Unidata WWW Service               http://www.unidata.ucar.edu/ *
+----------------------------------------------------------------+


On Wed, Mar 29, 2023 at 10:00 AM Herzmann, Daryl E [AGRON] 
<akrherz@xxxxxxxxxxx<mailto:akrherz@xxxxxxxxxxx>> wrote:
Howdy Gilbert,

I'm gonna sound like a grump again, but we need to be very careful about 
putting EMWIN data onto the IDD feed.  The reason is that the EMWIN and 
NOAAPort feeds are not byte-by-byte identical, so LDM has no chance to properly 
dedup these products.

For example, the ADMSDM product was duplicated twice:

      wmo_valid_at      |                product_origin                 |  
seqnum  | size
------------------------+-----------------------------------------------+----------+------
 2023-03-29 13:54:00+00 | 
exo.aggregatehail.com_v_idd.agron.iastate.edu<http://exo.aggregatehail.com_v_idd.agron.iastate.edu>
 |      999 |  742
 2023-03-29 13:54:00+00 | 
np2.ssec.wisc.edu_v_idd.agron.iastate.edu<http://np2.ssec.wisc.edu_v_idd.agron.iastate.edu>
     | 73462429 |  743
 2023-03-29 14:07:00+00 | 
exo.aggregatehail.com_v_idd.agron.iastate.edu<http://exo.aggregatehail.com_v_idd.agron.iastate.edu>
 |      999 |  742
 2023-03-29 14:07:00+00 | 
leno.unidata.ucar.edu_v_idd.agron.iastate.edu<http://leno.unidata.ucar.edu_v_idd.agron.iastate.edu>
 | 73472992 |  743
(4 rows)

As a reminder, the LDM MD5 checksum ignores the first 11 bytes of a product, so 
the "sequence number" at the top of the product does not matter for the MD5 
calculation.  The most common difference between the two feeds is the presence 
or absence of CR or LF at the end of the product, but this does not account for 
all of the differences.  Another difference is the "Aviation Control Character" 
that gets stripped out.

In this case, and importantly to whomever wrote the EMWIN to IDD ingest, is the 
missing space character after the sequence number:

999^M^M  <-- missing space here
NOUS42 KWNO 291407^M^M
ADMSDM^M^M

This space needs to be there for the 11 byte ignore calculation to be correct.

It would be nice that we could turn on this level of redundancy full-time, but 
we'd likely need to condition the noaaport + EMWIN ingests to rectify these 
differences so that LDM can generate identical MD5 and properly dedup the feeds.

daryl

________________________________________
From: ldm-users 
<ldm-users-bounces@xxxxxxxxxxxxxxxx<mailto:ldm-users-bounces@xxxxxxxxxxxxxxxx>> 
on behalf of Sebenste, Gilbert <sebensteg@xxxxxxx<mailto:sebensteg@xxxxxxx>>
Sent: Wednesday, March 29, 2023 9:18 AM
To: Mike Zuranski
Cc: LDM; NOAAPORT
Subject: Re: [ldm-users] [External] Re: [noaaport] NOAAport/SBN outage as of 
13:30Z

AllisonHouse just turned on a kludged EMWIN feed to at least get some 
IDS|DDPLUS products going again until this outage is over with. They are 
sending it to UNIDATA, so they hope it is getting through the system.

Gilbert Sebenste
Meteorology Support Analyst

[cid:image001.png@01D9621F.5C34C250]

From: Mike Zuranski <mzuranski@xxxxxxxx<mailto:mzuranski@xxxxxxxx>>
Sent: Wednesday, March 29, 2023 8:44 AM
To: Sebenste, Gilbert <sebensteg@xxxxxxx<mailto:sebensteg@xxxxxxx>>
Cc: LDM <ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx>>; 
NOAAPORT <noaaport@xxxxxxxxxxxxxxxx<mailto:noaaport@xxxxxxxxxxxxxxxx>>
Subject: [External] Re: [noaaport] NOAAport/SBN outage as of 13:30Z

CAUTION: This email originated from outside of COD’s system. Do not click 
links, open attachments, or respond with sensitive information unless you 
recognize the sender and know the content is safe.

Morning Gilbert,

I'm seeing the same and am investigating.  Thanks for your report.

Best,
-Mike


Mike Zuranski
Data Engineer II
Unidata Program Center
University Corporation for Atmospheric Research


On Wed, Mar 29, 2023 at 8:43 AM Sebenste, Gilbert 
<sebensteg@xxxxxxx<mailto:sebensteg@xxxxxxx><mailto:sebensteg@xxxxxxx<mailto:sebensteg@xxxxxxx>>>
 wrote:
Getting a strong NOAAport/SBN signal here at the College of DuPage, but outside 
of literally 6 products since 13:30Z, the feed has been dead.

This as of 13:42Z.

Gilbert

Gilbert Sebenste
Meteorology Support Analyst
[cid:image001.png@01D9621F.5C34C250]

_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web.  Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.


noaaport mailing list
noaaport@xxxxxxxxxxxxxxxx<mailto:noaaport@xxxxxxxxxxxxxxxx><mailto:noaaport@xxxxxxxxxxxxxxxx<mailto:noaaport@xxxxxxxxxxxxxxxx>>
For list information or to unsubscribe, visit: 
https://www.unidata.ucar.edu/mailing_lists/
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web.  Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.


ldm-users mailing list
ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx>
For list information or to unsubscribe,  visit: 
https://www.unidata.ucar.edu/mailing_lists/
  • 2023 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the ldm-users archives: