I disagree. And to illustrate, this is what happens when you run the emwin
insertion:
20230329T145036.073685Z pqinsert[2687] pqinsert.c:main:443
ERROR Product already in queue:
1dc1a8be4c16707deb53b8b54c91e9d2 89 20230329145036.073608 DDPLUS
000 SAUS80 KWBC 291500 /pMETAR
20230329T145036.240065Z pqinsert[2939] pqinsert.c:main:443
ERROR Product already in queue:
4ec193533d40792d288ad39669779d86 1299 20230329145036.239987 DDPLUS
000 FPUS63 KBIS 291447 /pSFPND
20230329T145036.481866Z pqinsert[3302] pqinsert.c:main:443
ERROR Product already in queue:
6466efe262a46bd5941ab7fdf92b77ab 963 20230329145036.481785 DDPLUS
000 SPUS70 KWBC 291446 /pSPECI
20230329T145036.642540Z pqinsert[3545] pqinsert.c:main:443
ERROR Product already in queue:
b630c518ccbcd090b5dbb853eeec232c 242 20230329145036.642460 DDPLUS
000 SAUS80 KWBC 291446 /pMETAR
20230329T145036.727266Z pqinsert[3671] pqinsert.c:main:443
ERROR Product already in queue:
ffbbda56d34ae475e4da727428f76c98 413 20230329145036.727189 DDPLUS
000 SRUS43 KKRF 291446 /pRVMKRF
20230329T145037.245965Z pqinsert[4474] pqinsert.c:main:443
ERROR Product already in queue:
3839d746854a538f956433100b581f20 162 20230329145037.245882 DDPLUS
000 NOUS64 KMOB 291447 /pFTMMOB
20230329T145037.769953Z pqinsert[5261] pqinsert.c:main:443
ERROR Product already in queue:
6e418199e50d54393474f1bbf6d0d8e0 2008 20230329145037.769858 DDPLUS
000 FLUS43 KBIS 291446 /pHWOBIS
20230329T145037.890334Z pqinsert[5443] pqinsert.c:main:443
ERROR Product already in queue:
0486d0d6331694e523d4ff36d6d6f2db 3373 20230329145037.890259 DDPLUS
000 ASUS65 KABQ 291447 /pRTPNM
20230329T145038.180060Z pqinsert[5870] pqinsert.c:main:443
ERROR Product already in queue:
8e183a08bdf5c3f51937787a20521313 2514 20230329145038.179980 DDPLUS
000 SAUS70 KWBC 291446 /pMETAR
The deduplication nature of the LDM will filter out the second copy. In
the end, having a more resilient decoding apparatus is more important than
possibly getting duplicates as compared to none.
Stonie
+----------------------------------------------------------------+
* Stonie Cooper UCAR Unidata Program *
* Software Engineer III P.O. Box 3000 *
* cooper@xxxxxxxx Boulder, CO 80307 *
* Unidata WWW Service http://www.unidata.ucar.edu/ *
+----------------------------------------------------------------+
On Wed, Mar 29, 2023 at 10:00 AM Herzmann, Daryl E [AGRON] <
akrherz@xxxxxxxxxxx> wrote:
> Howdy Gilbert,
>
> I'm gonna sound like a grump again, but we need to be very careful about
> putting EMWIN data onto the IDD feed. The reason is that the EMWIN and
> NOAAPort feeds are not byte-by-byte identical, so LDM has no chance to
> properly dedup these products.
>
> For example, the ADMSDM product was duplicated twice:
>
> wmo_valid_at | product_origin |
> seqnum | size
>
> ------------------------+-----------------------------------------------+----------+------
> 2023-03-29 13:54:00+00 | exo.aggregatehail.com_v_idd.agron.iastate.edu
> | 999 | 742
> 2023-03-29 13:54:00+00 | np2.ssec.wisc.edu_v_idd.agron.iastate.edu |
> 73462429 | 743
> 2023-03-29 14:07:00+00 | exo.aggregatehail.com_v_idd.agron.iastate.edu
> | 999 | 742
> 2023-03-29 14:07:00+00 | leno.unidata.ucar.edu_v_idd.agron.iastate.edu |
> 73472992 | 743
> (4 rows)
>
> As a reminder, the LDM MD5 checksum ignores the first 11 bytes of a
> product, so the "sequence number" at the top of the product does not matter
> for the MD5 calculation. The most common difference between the two feeds
> is the presence or absence of CR or LF at the end of the product, but this
> does not account for all of the differences. Another difference is the
> "Aviation Control Character" that gets stripped out.
>
> In this case, and importantly to whomever wrote the EMWIN to IDD ingest,
> is the missing space character after the sequence number:
>
> 999^M^M <-- missing space here
> NOUS42 KWNO 291407^M^M
> ADMSDM^M^M
>
> This space needs to be there for the 11 byte ignore calculation to be
> correct.
>
> It would be nice that we could turn on this level of redundancy full-time,
> but we'd likely need to condition the noaaport + EMWIN ingests to rectify
> these differences so that LDM can generate identical MD5 and properly dedup
> the feeds.
>
> daryl
>
> ________________________________________
> From: ldm-users <ldm-users-bounces@xxxxxxxxxxxxxxxx> on behalf of
> Sebenste, Gilbert <sebensteg@xxxxxxx>
> Sent: Wednesday, March 29, 2023 9:18 AM
> To: Mike Zuranski
> Cc: LDM; NOAAPORT
> Subject: Re: [ldm-users] [External] Re: [noaaport] NOAAport/SBN outage as
> of 13:30Z
>
> AllisonHouse just turned on a kludged EMWIN feed to at least get some
> IDS|DDPLUS products going again until this outage is over with. They are
> sending it to UNIDATA, so they hope it is getting through the system.
>
> Gilbert Sebenste
> Meteorology Support Analyst
>
> [cid:image001.png@01D9621F.5C34C250]
>
> From: Mike Zuranski <mzuranski@xxxxxxxx>
> Sent: Wednesday, March 29, 2023 8:44 AM
> To: Sebenste, Gilbert <sebensteg@xxxxxxx>
> Cc: LDM <ldm-users@xxxxxxxxxxxxxxxx>; NOAAPORT <noaaport@xxxxxxxxxxxxxxxx>
> Subject: [External] Re: [noaaport] NOAAport/SBN outage as of 13:30Z
>
> CAUTION: This email originated from outside of COD’s system. Do not click
> links, open attachments, or respond with sensitive information unless you
> recognize the sender and know the content is safe.
>
> Morning Gilbert,
>
> I'm seeing the same and am investigating. Thanks for your report.
>
> Best,
> -Mike
>
>
> Mike Zuranski
> Data Engineer II
> Unidata Program Center
> University Corporation for Atmospheric Research
>
>
> On Wed, Mar 29, 2023 at 8:43 AM Sebenste, Gilbert <sebensteg@xxxxxxx
> <mailto:sebensteg@xxxxxxx>> wrote:
> Getting a strong NOAAport/SBN signal here at the College of DuPage, but
> outside of literally 6 products since 13:30Z, the feed has been dead.
>
> This as of 13:42Z.
>
> Gilbert
>
> Gilbert Sebenste
> Meteorology Support Analyst
> [cid:image001.png@01D9621F.5C34C250]
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> noaaport mailing list
> noaaport@xxxxxxxxxxxxxxxx<mailto:noaaport@xxxxxxxxxxxxxxxx>
> For list information or to unsubscribe, visit:
> https://www.unidata.ucar.edu/mailing_lists/
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> ldm-users mailing list
> ldm-users@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> https://www.unidata.ucar.edu/mailing_lists/
>
