Hi Gilbert and other ldm-users,
re:
>I have questions about this transition, and I'm confused about a few
>things, so hopefully this can be cleared up.
OK, ready...
re:
>First: Does this affect anything on the SATELLITE/nee DIFAX feed?
No.
re:
>That is
>strictly for data coming across the GRB feed and will stay there, correct?
Correct.
re:
>Second: What about those who have their own ingest dishes? If they use IDD
>as a backup, will they need to have a duplicate feed coming across: one
>from their dish on NOTHER/EXP, and one from UNIDATA? If so...
The NOAAPort ingest package that is bundled with the LDM has not been
changed. The contents of the NOTHER and HDS datastreams have not been
altered. We are stitching together the ABI L2 image tiles and stripping
off the NOAAPort headers and footers on a non-ingest machine.
re:
>Third: Does this require an LDM or LDM database upgrade, the latter of
>which can be done automagically in the latest version, 6.13.11?
No.
re:
>Fourth: Can the LDM switch the NOAAport-ingested GOES products over to
>NIMAGE as well to avoid duplicating the UNIDATA feed on the current
>feedtypes IF a site has their own dish/receiver?
I am not sure that I fully understand this question.
For the time being, the NOAAPort ingest capability that is packaged with
the LDM will continue to classify the ABI L2 tiles and some of the other
L2 products in the NOTHER datastream. The other L2 products that are in
the HDS feed will continue to be in the HDS feed. We realize that this
split between two different feeds for the unaltered, NOAAPort-delivered
products should be addressed, but this will have to wait as there are
lots of more critical items that need attention.
re:
>Thank you for the clarification in advance,
No worries.
Cheers,
Tom
>On Tue, Jun 4, 2019 at 12:51 PM Unidata User Support <
>support@xxxxxxxxxxxxxxxx> wrote:
>
>> IMPORTANT notice of major changes to the IDD NIMAGE datastream; please
>> read.
>>
>> First, we apologies for cross posting this announcement to multiple lists,
>> but we want to make sure that all users that will/could be affected
>> are notified.
>>
>> Second, we realize that the following message is long, but please read
>> it all since it contains important information that will affect sites
>> that are or will be receiving imagery products in the IDD NIMAGE feed.
>>
>> What:
>>
>> We will be transitioning the IDD NIMAGE datastream to one that
>> contains all of the GOES-16 and GOES-17 Level 2 image products
>> that are currently being being sent in the IDD NOTHER and HDS
>> datastreams.
>>
>> When:
>>
>> It is our desire to begin distributing a reconstituted NIMAGE feed
>> from our top level IDD relay clusters (idd.unidata.ucar.edu and
>> iddb.unidata.ucar.edu) in the mid-June time frame. We will postpone
>> turning on this feed if/when end-users indicate that they need more
>> time to prepare for the change.
>>
>> Impact:
>>
>> The volume of data available in the NIMAGE feed will increase from
>> an average of about 100 MB/hr to an average that is close to 7 GB/hr,
>> a 70-fold increase!
>>
>> Reason for the change:
>>
>> NOAA has indicated that operation of GOES-15 will likely cease in
>> early-mid July:
>>
>> https://www.goes-r.gov/users/transitionToOperations17.html
>>
>> Since GOES-East imagery in GINI format was removed from the NOAAPort
>> SBN after GOES-13 was replaced as GOES-East by GOES-16 and then
>> decommissioned, we believe that it is most likely that all GINI imagery
>> will cease to be distributed in NOAAPort in the mid-July time frame.
>> Since the NIMAGE IDD data feed has been populated solely by the GINI
>> imagery distributed in NOAAPort, this feed will cease to contain any data.
>>
>> Currently, NOAAPort contains GOES-16 and GOES-17 ABI (Advanced Baseline
>> Imager) Level 2 product tiles (pieces of full image scenes), and those
>> tiles are being distributed in the IDD NOTHER data feed. To date, these
>> image tiles would need to be stitched together by end-users before they
>> were fully usable in display and analysis applications available from
>> Unidata (e.g., AWIPS, GEMPAK, IDV, McIDAS and MetPy (this is not
>> strictly true for AWIPS)).
>>
>> In addition to the ABI Level 2 product tiles, 19 other Level 2 products
>> are also being distributed in NOAAPort, and we have been making those
>> products available in the IDD NOTHER and HDS data feeds since their
>> addition to NOAAPort. Like the ABI Level 2 product tiles, the other
>> Level 2 products need some processing before they are usable in end-user
>> applications - they need to have the NOAAPort header and footer stripped
>> off to leave the underlying netCDF4 files.
>>
>> We believe that it will be much easier for end-users, who get their
>> NOAAPort-delivered Level 2 images and products via the IDD, if we
>> reconstitute full images scenes from ABI tiles and strip NOAAPort
>> broadcast headers and footers here in Unidata and then distributed the
>> resulting netCDF4 images/products in a reconstituted NIMAGE IDD data
>> feed. Also, reconstituting the NIMAGE feed will allow us to make other,
>> value-added GOES-16/17 products available to the community. The first
>> of these products is GLM imagery/grids being produced at Texas Tech by
>> community member Eric Bruning.
>>
>> Information on the imagery and products that are currently contained
>> in and those that will be added to the reconstituted NIMAGE feed can be
>> found in:
>>
>> https://www.unidata.ucar.edu/data/nimage.html
>>
>> We have been using all of the products in this new incarnation of the
>> NIMAGE feed internally for the past few months to test their viability
>> in the various application packages that we provide and support. We are
>> satisfied that transition to use of these products by end users will be
>> straightforward, but it will require that sites install and transition
>> to use of the latest of each package that we make available.
>>
>> Important things to note:
>>
>> - the reconstitution of the NIMAGE feed comes at a cost
>>
>> The volume of data available will increase from an average of about
>> 100 MB/hr to an average nearing 7 GB/hr, a 70-fold increase!
>>
>> Sites that do not want to receive any/all of the processed GOES-16/17
>> imagery/products in the new NIMAGE feed need to make sure that their
>> LDM configuration file REQUEST(s) do not include products that they
>> do not want.
>>
>> The current GOES-15 GINI imagery that is being distributed in NOAAPort
>> all have LDM/IDD Product IDs that begin with 'satz'. Sites requesting
>> all of the current NIMAGE feed via a REQUEST that looks like:
>>
>> REQUEST NIMAGE ".*" idd.unidata.ucar.edu
>>
>> and that either do not want or can not handle the increased data volume
>> should immediately change their REQUEST to something like:
>>
>> REQUEST NIMAGE "satz" idd.unidata.ucar.edu
>>
>> Sites that do want to receive all of the imagery that will be available
>> in the reconstituted NIMAGE feed and whose network connection can handle
>> the increased volume can leave their REQUEST(s) as is, but this decision
>> should not be taken lightly given the 70-fold increase in data volume!
>>
>> Sites wanting to receive some of the data from the reconstituted
>> NIMAGE feed are advised to read through the NIMAGE web page listed
>> above to become familiar with how the LDM/IDD Product IDs will look
>> and then re-fashion their feed REQUEST(s) to get the set of products
>> desired.
>>
>> - sites that want to use the products that will be available in the
>> reconstituted NIMAGE feed should immediately install and switch to
>> using the most recent releases of their Unidata-supplied display and
>> analysis applications
>>
>> This is especially true for GEMPAK users as imagery in netCDF4 format
>> was not supported in Unidata GEMPAK prior to the current release.
>>
>> - the data volume that will be available in the reconstituted NIMAGE
>> feed is actually 20% less than the volume of the sum of the products
>> being delivered in NOAAPort and distributed in the NOTHER and HDS
>> datastreams
>>
>> The reason for this is twofold:
>>
>> - NOAAPort broadcast headers and footers have been removed
>> resulting in a minor reduction of volume
>>
>> - netCDF4 compression has been implemented/implemented better in
>> the products
>>
>> Recap:
>>
>> The reconstituted IDD NIMAGE datastream will contain:
>>
>> - full image scenes that have been created by stitching together
>> the ABI tiles sent in the NOAAPort SBN and distributed in the
>> IDD NOTHER feed
>>
>> - all other Level 2 products being distributed in the NOAAPort SBN
>> in the IDD NOTHER and HDS feeds
>>
>> - value added GLM image/grid products created by Eric Bruning
>> of Texas Tech University
>>
>> - all products are in netCDF-4 format
>>
>> - all products are in the GOES-R mission standard Fixed Grid
>> projection
>>
>> - the names for all products (contained in the LDM/IDD Product IDs)
>> follow the GOES-R mission standard naming convention as detailed
>> in:
>>
>> GOES-R SERIES PRODUCT DEFINITION AND USERS=E2=80=99 GUIDE (PUG)
>> https://www.goes-r.gov/users/docs/PUG-GRB-vol4.pdf
>>
>> In all cases, the LDM/IDD Product IDs will consist of fully qualified
>> path names for products, and the file name portion of the fully
>> qualified names follow GOES-R mission standards for naming. The NIMAGE
>> web page referenced above lists example Product IDs for each product.
>>
>> Please send any/all questions/comments about the upcoming change to the
>> IDD NIMAGE feed to:
>>
>> support-datastream@xxxxxxxxxxxxxxxx
>>
>> Best regards,
>>
>> Unidata User Support
--
********************************************************************** <
Unidata User Support UCAR Unidata Program <
(303)497-8643 P.O. Box 3000 <
support@xxxxxxxxxxxxxxxx Boulder, CO 80307 <
---------------------------------------------------------------------- <
Unidata Web Support http://www.unidata.ucar.edu/support <
---------------------------------------------------------------------- <
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publicly available
through the web. If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.
