[noaaport] The SBN Outage, Supplemental IDD Data and Duplicate Products

Greetings everyone,
 
Last Wednesday a NOAAPort outage sparked a conversation on these lists regarding the insertion of products into the IDD, specifically the EMWIN feed in this case, and the prevention of duplicate products being inserted.  We had an internal discussion afterwards last week and we want to give you all an update.  Below is a brief rundown on what happened and what we are proposing moving forward.
 
After data stopped flowing over NOAAPort, products from the EMWIN feed were inserted into the IDD as a stop-gap measure to restore at least some of the IDD|DDPLUS feed.  However, the script used was rather old and didn’t fully handle the differences between EMWIN and NOAAPort data that exist today.  As a result, some duplicate products were inserted into the IDD as data began to flow over NOAAPort again.
 
In the discussions that followed we believe the technical details in improving that EMWIN insertion script aren’t too involved; trim some trailing whitespaces which may or may not be present, include key parts of the WMO header while ignoring others (e.g. the three-digit product sequence number), and the duplicate product rejection logic in LDM ought to handle this situation moving forward.  We are working on proofing these changes now.
 
We are also investigating how we can best make this switchover automatic here at Unidata.  In the event of another NOAAPort outage, we are proposing to insert supplemental data from sources such as EMWIN from our end to compensate.  With the improved handling mentioned above we should be able to ensure no duplicate products will be entered into the IDD, and with this operation running out of the Unidata Program Center we’ll be able to quickly address any unexpected issues as we deploy this capability.  This may take some time to implement, no ETA at this time but we believe this is the best path forward.
 
We’d again like to thank everyone for contributing their input last Wednesday, it was very beneficial to help drive our internal discussion.  We’ll continue to update as we move forward on this, but please feel free to reach out with any questions or concerns.
 
Best,
-Mike

_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web.  Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.


noaaport mailing list
address@hidden
For list information or to unsubscribe, visit: 
https://www.unidata.ucar.edu/mailing_lists/