Service Update: Changes to the IDD NIMAGE Datastream

The Unidata Program Center will be altering the IDD NIMAGE datastream to contain all of the GOES-16 and GOES-17 Level 2 image products that are currently being being sent in the IDD NOTHER and HDS datastreams.

We plan to begin distributing a reconstituted NIMAGE feed from our top level IDD relay clusters (idd.unidata.ucar.edu and iddb.unidata.ucar.edu) in the mid-June time frame. (Note: We will postpone turning on this feed if/when end users indicate that they need more time to prepare for the change.)

This is important because the volume of data available in the NIMAGE feed will increase from an average of about 100 MB/hr to an average that is close to 7 GB/hr, a 70-fold increase!

Why is Unidata Making This Change?

NOAA has indicated that operation of GOES-15 will likely cease in early-mid July:

https://www.goes-r.gov/users/transitionToOperations17.html

Since GOES-East imagery in GINI format was removed from the NOAAPort SBN after GOES-13 was replaced by GOES-16 and then decommissioned, we believe that it is most likely that all GINI imagery will cease to be distributed in NOAAPort in the mid-July time frame. Since the NIMAGE IDD data feed has been populated solely by the GINI imagery distributed in NOAAPort, this feed will cease to contain any data.

Currently, NOAAPort contains GOES-16 and GOES-17 ABI (Advanced Baseline Imager) Level 2 product tiles (pieces of full image scenes), and those tiles are being distributed in the IDD NOTHER data feed. To date, these image tiles would need to be stitched together by end users before they were fully usable in display and analysis applications available from Unidata (e.g., AWIPS, GEMPAK, IDV, McIDAS and MetPy).

In addition to the ABI Level 2 product tiles, 19 other Level 2 products are also being distributed in NOAAPort. Unidata has been making those products available in the IDD NOTHER and HDS data feeds since their addition. Like the ABI Level 2 product tiles, the other Level 2 products need some processing before they are usable in end user applications — they need to have the NOAAPort header and footer stripped off to leave the underlying netCDF4 files.

We believe that it will be much easier for end users who get their NOAAPort-delivered Level 2 images and products via the IDD if we reconstitute full images scenes from ABI tiles and strip NOAAPort broadcast headers and footers here at the Unidata Program Center and then distribute the resulting netCDF4 images/products in a reconstituted NIMAGE IDD data feed. Also, reconstituting the NIMAGE feed will allow us to make other, value-added GOES-16/17 products available to the community. The first of these products is Geostationary Lightning Mapper (GLM) imagery/grids being produced at Texas Tech by community member Eric Bruning.

Information on the imagery and products that are currently contained in and those that will be added to the reconstituted NIMAGE feed can be found in:

https://www.unidata.ucar.edu/data/nimage.html

We have been using all of the products in this new incarnation of the NIMAGE feed internally for the past few months to test their viability in the various application packages that we provide and support. We are satisfied that transition to use of these products by end users will be straightforward, but it will require that sites install and transition to use of the latest of each package that we make available.

Important Things to Note

1. The reconstitution of the NIMAGE feed comes at a cost

The volume of data available will increase from an average of about 100 MB/hr to an average nearing 7 GB/hr, a 70-fold increase!

Sites that do not want to receive any/all of the processed GOES-16/17 imagery/products in the new NIMAGE feed need to make sure that their LDM configuration file REQUEST(s) do not include products that they do not want.

The current GOES-15 GINI images being distributed in NOAAPort all have LDM/IDD Product IDs that begin with 'satz'. Sites requesting all of the current NIMAGE feed via a REQUEST that looks like:

REQUEST NIMAGE ".*" idd.unidata.ucar.edu

and that either do not want or can not handle the increased data volume should immediately change their REQUEST to something like:

REQUEST NIMAGE "satz" idd.unidata.ucar.edu

Sites that do want to receive all of the imagery that will be available in the reconstituted NIMAGE feed and whose network connection can handle the increased volume can leave their REQUEST(s) as is, but this decision should not be taken lightly given the 70-fold increase in data volume.

Sites wanting to receive some of the data from the reconstituted NIMAGE feed are advised to read through the NIMAGE web page listed above to become familiar with how the LDM/IDD Product IDs will look and then re-fashion their feed REQUEST(s) to get the set of products desired.

In addition, sites wanting to use some/all of the data in the new incarnation of the NIMAGE feed will need to modify/add LDM pattern-action file action(s) to store the products for use in the display and analysis package(s) that they use. The NIMAGE web page contains links to example pattern-action file actions that illustrate one approach for supporting the new imagery. Sites that have questions about how they should modify their existing LDM setup should contact Unidata User Support for assistance (see below).

2. Sites must upgrade visualization applications

Sites that want to use the products that will be available in the reconstituted NIMAGE feed should immediately install and switch to using the most recent releases of their Unidata-supplied display and analysis applications

This is especially true for GEMPAK users, as imagery in netCDF4 format was not supported in Unidata GEMPAK prior to the current release.

3. Data volumes are being minimized

The volume of the data that will be available in the reconstituted NIMAGE feed is actually 20% less than the volume of the sum of the products being delivered in NOAAPort and distributed in the NOTHER and HDS datastreams. The reason for this is twofold:

  • NOAAPort broadcast headers and footers have been removed resulting in a minor reduction of volume
  • netCDF4 compression has been implemented/implemented better in the products

In Summary

The reconstituted IDD NIMAGE datastream will contain:

  • Full image scenes that have been created by stitching together the ABI tiles sent in the NOAAPort SBN and distributed in the IDD NOTHER feed
  • All other Level 2 products being distributed in the NOAAPort SBN in the IDD NOTHER and HDS feeds
  • Value-added GLM image/grid products created by Eric Bruning of Texas Tech University

Note also:

  • All products are in netCDF-4 format
  • All products are in the GOES-R mission standard Fixed Grid projection
  • The names for all products (contained in the LDM/IDD Product IDs) follow the GOES-R mission standard naming convention as detailed in the
    GOES-R SERIES PRODUCT DEFINITION AND USERS’ GUIDE (PUG).
  • In all cases, the LDM/IDD Product IDs will consist of fully qualified path names for products, and the file name portion of the fully qualified names follow GOES-R mission standards for naming. The NIMAGE web page referenced above lists example Product IDs for each product.

Please send questions and comments about the change to the IDD NIMAGE feed to: support-datastream@unidata.ucar.edu

Comments:

Post a Comment:
  • HTML Syntax: Allowed
News@Unidata
News and information from the Unidata Program Center
News@Unidata
News and information from the Unidata Program Center

Welcome

FAQs

Developers’ blog

Take a poll!

What if we had an ongoing user poll in here?

Browse By Topic
Browse by Topic
« July 2019
SunMonTueWedThuFriSat
 
3
4
5
6
7
8
10
11
12
13
14
15
16
17
18
20
21
22
23
24
25
26
27
28
29
30
31
   
       
Today