Re: [ldm-users] 20190604: major upcoming change to IDD NIMAGE datastream

Hello Tom,

I have questions about this transition, and I'm confused about a few
things, so hopefully this can be cleared up.

First: Does this affect anything on the SATELLITE/nee DIFAX feed? That is
strictly for data coming across the GRB feed and will stay there, correct?

Second: What about those who have their own ingest dishes? If they use IDD
as a backup, will they need to have a duplicate feed coming across: one
from their dish on NOTHER/EXP, and one from UNIDATA? If so...

Third: Does this require an LDM or LDM database upgrade, the latter of
which can be done automagically in the latest version, 6.13.11?

Fourth: Can the LDM switch the NOAAport-ingested GOES products over to
NIMAGE as well to avoid duplicating the UNIDATA feed on the current
feedtypes IF a site has their own dish/receiver?

Thank you for the clarification in advance,

Gilbert Sebenste,

Data guru, AllisonHouse, LLC

On Tue, Jun 4, 2019 at 12:51 PM Unidata User Support <
support@xxxxxxxxxxxxxxxx> wrote:

> IMPORTANT notice of major changes to the IDD NIMAGE datastream; please
> read.
>
> First, we apologies for cross posting this announcement to multiple lists,
> but we want to make sure that all users that will/could be affected
> are notified.
>
> Second, we realize that the following message is long, but please read
> it all since it contains important information that will affect sites
> that are or will be receiving imagery products in the IDD NIMAGE feed.
>
> What:
>
> We will be transitioning the IDD NIMAGE datastream to one that
> contains all of the GOES-16 and GOES-17 Level 2 image products
> that are currently being being sent in the IDD NOTHER and HDS
> datastreams.
>
> When:
>
> It is our desire to begin distributing a reconstituted NIMAGE feed
> from our top level IDD relay clusters (idd.unidata.ucar.edu and
> iddb.unidata.ucar.edu) in the mid-June time frame.  We will postpone
> turning on this feed if/when end-users indicate that they need more
> time to prepare for the change.
>
> Impact:
>
> The volume of data available in the NIMAGE feed will increase from
> an average of about 100 MB/hr to an average that is close to 7 GB/hr,
> a 70-fold increase!
>
> Reason for the change:
>
> NOAA has indicated that operation of GOES-15 will likely cease in
> early-mid July:
>
> https://www.goes-r.gov/users/transitionToOperations17.html
>
> Since GOES-East imagery in GINI format was removed from the NOAAPort
> SBN after GOES-13 was replaced as GOES-East by GOES-16 and then
> decommissioned, we believe that it is most likely that all GINI imagery
> will cease to be distributed in NOAAPort in the mid-July time frame.
> Since the NIMAGE IDD data feed has been populated solely by the GINI
> imagery distributed in NOAAPort, this feed will cease to contain any data .
>
> Currently, NOAAPort contains GOES-16 and GOES-17 ABI (Advanced Baseline
> Imager) Level 2 product tiles (pieces of full image scenes), and those
> tiles are being distributed in the IDD NOTHER data feed.  To date, these
> image tiles would need to be stitched together by end-users before they
> were fully usable in display and analysis applications available from
> Unidata (e.g., AWIPS, GEMPAK, IDV, McIDAS and MetPy (this is not
> strictly true for AWIPS)).
>
> In addition to the ABI Level 2 product tiles, 19 other Level 2 products
> are also being distributed in NOAAPort, and we have been making those
> products available in the IDD NOTHER and HDS data feeds since their
> addition to NOAAPort.  Like the ABI Level 2 product tiles, the other
> Level 2 products need some processing before they are usable in end-user
> applications - they need to have the NOAAPort header and footer stripped
> off to leave the underlying netCDF4 files.
>
> We believe that it will be much easier for end-users, who get their
> NOAAPort-delivered Level 2 images and products via the IDD, if we
> reconstitute full images scenes from ABI tiles and strip NOAAPort
> broadcast headers and footers here in Unidata and then distributed the
> resulting netCDF4 images/products in a reconstituted NIMAGE IDD data
> feed.  Also, reconstituting the NIMAGE feed will allow us to make other,
> value-added GOES-16/17 products available to the community.  The first
> of these products is GLM imagery/grids being produced at Texas Tech by
> community member Eric Bruning.
>
> Information on the imagery and products that are currently contained
> in and those that will be added to the reconstituted NIMAGE feed can be
> found in:
>
> https://www.unidata.ucar.edu/data/nimage.html
>
> We have been using all of the products in this new incarnation of the
> NIMAGE feed internally for the past few months to test their viability
> in the various application packages that we provide and support.  We are
> satisfied that transition to use of these products by end users will be
> straightforward, but it will require that sites install and transition
> to use of the latest of each package that we make available.
>
> Important things to note:
>
> - the reconstitution of the NIMAGE feed comes at a cost
>
>   The volume of data available will increase from an average of about
>   100 MB/hr to an average nearing 7 GB/hr, a 70-fold increase!
>
>   Sites that do not want to receive any/all of the processed GOES-16/17
>   imagery/products in the new NIMAGE feed need to make sure that their
>   LDM configuration file REQUEST(s) do not include products that they
>   do not want.
>
>   The current GOES-15 GINI imagery that is being distributed in NOAAPort
>   all have LDM/IDD Product IDs that begin with 'satz'.  Sites requesting
>   all of the current NIMAGE feed via a REQUEST that looks like:
>
>   REQUEST NIMAGE ".*" idd.unidata.ucar.edu
>
>   and that either do not want or can not handle the increased data volume
>   should immediately change their REQUEST to something like:
>
>   REQUEST NIMAGE "satz" idd.unidata.ucar.edu
>
>   Sites that do want to receive all of the imagery that will be available
>   in the reconstituted NIMAGE feed and whose network connection can handle
>   the increased volume can leave their REQUEST(s) as is, but this decision
>   should not be taken lightly given the 70-fold increase in data volume!
>
>   Sites wanting to receive some of the data from the reconstituted
>   NIMAGE feed are advised to read through the NIMAGE web page listed
>   above to become familiar with how the LDM/IDD Product IDs will look
>   and then re-fashion their feed REQUEST(s) to get the set of products
>   desired.
>
> - sites that want to use the products that will be available in the
>   reconstituted NIMAGE feed should immediately install and switch to
>   using the most recent releases of their Unidata-supplied display and
>   analysis applications
>
>   This is especially true for GEMPAK users as imagery in netCDF4 format
>   was not supported in Unidata GEMPAK prior to the current release.
>
> - the data volume that will be available in the reconstituted NIMAGE
>   feed is actually 20% less than the volume of the sum of the products
>   being delivered in NOAAPort and distributed in the NOTHER and HDS
>   datastreams
>
>   The reason for this is twofold:
>
>   - NOAAPort broadcast headers and footers have been removed
>     resulting in a minor reduction of volume
>
>   - netCDF4 compression has been implemented/implemented better in
>     the products
>
> Recap:
>
> The reconstituted IDD NIMAGE datastream will contain:
>
> - full image scenes that have been created by stitching together
>   the ABI tiles sent in the NOAAPort SBN and distributed in the
>   IDD NOTHER feed
>
> - all other Level 2 products being distributed in the NOAAPort SBN
>   in the IDD NOTHER and HDS feeds
>
> - value added GLM image/grid products created by Eric Bruning
>   of Texas Tech University
>
> - all products are in netCDF-4 format
>
> - all products are in the GOES-R mission standard Fixed Grid
>   projection
>
> - the names for all products (contained in the LDM/IDD Product IDs)
>   follow the GOES-R mission standard naming convention as detailed
>   in:
>
>   GOES-R SERIES PRODUCT DEFINITION AND USERS’ GUIDE (PUG)
>   https://www.goes-r.gov/users/docs/PUG-GRB-vol4.pdf
>
>   In all cases, the LDM/IDD Product IDs will consist of fully qualified
>   path names for products, and the file name portion of the fully
>   qualified names follow GOES-R mission standards for naming. The NIMAGE
>   web page referenced above lists example Product IDs for each product.
>
> Please send any/all questions/comments about the upcoming change to the
> IDD NIMAGE feed to:
>
> support-datastream@xxxxxxxxxxxxxxxx
>
> Best regards,
>
> Unidata User Support
> --
> ********************************************************************** <
> Unidata User Support                              UCAR Unidata Program <
> (303)497-8643                                            P.O. Box 3000 <
> support@xxxxxxxxxxxxxxxx                             Boulder, CO 80307 <
> ---------------------------------------------------------------------- <
> Unidata Web Support               http://www.unidata.ucar.edu/support  <
> ********************************************************************** <
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web.  Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> ldm-users mailing list
> ldm-users@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe,  visit:
> http://www.unidata.ucar.edu/mailing_lists/



-- 
----

Gilbert Sebenste
Consulting Meteorologist
AllisonHouse, LLC
  • 2019 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the ldm-users archives: