[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #JLN-510178]: a few general questions on the current use ot NetCDF4 from the GOES-R GS program

Hi Randy,

>           ....we and some of my peers exchanged emails with you
> in the past concerning NetCDF4 capabilities, etc.  I am supporting
> Harris Ciorporation in the development of the GOES-R ground system.
> 
> I had mentioned to you that the Government has required the use of
> NetCDF4.
> 
> I have a few general technical questions about how others have been
> using NetCDF4.  Firsxt I need to provide you some background.
> 
> Background
> 
> The GOES-R GS will be generating NetCDF4 product files that (1) span
> the western hemisphere or (2) are rectangular regions within the
> western hemuisphere.
> 
> The product data is either (1) gridded or (2) a set of records.
> 
> Our general thinking is that the gridded data sets will be organized
> into "tiles" that either cover the hemisphere or rectangular regions.
> For example, these tiles might be 1024 x 1024 rectangular regions that
> are mosaiced together to cover the hemisphere.  We may decide to make
> the tile sizes configurable.
> 
> The size of our products is generally in the 100s of MBytes.
> 
> Not all the gridded products cover the entire area covered by the
> western hemisphere or the rectangular regions.  For example, some
> product data is only generated on land or sea, some during day or
> night, and some where there is no clouds.
> 
> It is advantageous for our system and our data user communities to
> have NetCDF4 product files that only contain tiles with valid data.
> 
> It is advantageous for our data user communities to be able to read
> specific subsets of our netCDF4 product files that correspond to
> specific regions. For example, a user may wish to display only the
> continental US region of a hemispheric product.
> 
> Questions
> 
> (1)
> 
> Is there a present producer of NetCDF files that generates product
> files containing tiles where the set of tiles in the product file is
> not in of itself a rectangular region (for example, tiles over sea
> only) ?

There are certainly products of ocean-only data, for example the
GHRSST datasets for Global High-Resolution Sea Surface Temperature
archived at the PO.DAAC at JPL .  Many ocean models generate gridded
outputs for ocean-only.  Some of these produce tiled data, but typically
for the whole globe.

Some GHRSST data is available as tiled swaths, and I think the L4 
gridded data is available as (KML?) tiles for Google Earth.

There are proposed CF conventions for tiling model outputs for CMIP5,
see the Gridspec proposal:

  https://cf-pcmdi.llnl.gov/trac/ticket/63

I think this is in regular use at GFDL for model output, but again
for global grids rather than regional grids.

There may be lots of other tiling schemes of which I'm not aware.

> (2)
> 
> If so, can you tell me the name of the application or originating
> agency ?

GFDL is under NOAA.  PCMDI at Lawrence Livermore is involved with the
Gridspec development, under DoE.  JPL and the PO.DAAC are NASA.

> (3)
> 
> Are there already metadata conventions that allow for a set of tiles
> to exist in a file that do not form a contiguous rectangular region ?

Good question, but I'm not sure I have a good answer.  The Gridspec
conventions provide general ways to stitch tiles together along common
boundaries to form non-rectangular regions.  There are unstructured
grid conventions under development, as part of a UGRID project, for
use in coastal and ocean models:

There is currently a Google Group on UGRID Interoperability that is
moving the UGRID proposal forward.  According to a developer who is a
member of that group, NOAA's IOOS is funding a modeling testbed to
implement things like subsetting and interpolation of unstructured
grids inside of netCDF-Java.

The NSF-funded Ocean Observatories Initiative has also published
drafts that mention netCDF and DAP representations for unstructured
grids.

> (4)
> 
> If we decide to not have lat/lon coordinates associated with each
> pixel (data point), are there established conventions for metadata
> that allow for relating geolocation information (i.e. lat/lon) with
> tiles. For example, the four lat/lon corner points of a tile).  Sort
> of like a geolocation mapping table where earth regions are mapped to
> specific tiles in the product file.

The CF Conventions require coordinates for each data point, but that
takes a lot of space.  When the mapping can be described as a
projection onto a plane, the CF conventions provide ways to represent
the projection parameters, but as far as I know there is no proposed 
convention for doing what you describe with just the corners.  I'm 
Cc:ing our local expert on this, John Caron, in case he knows more.

> (5)
> 
> Are there off the shelf NetCDF4 viewers that can deal with the type of
> product files being discussed here (i.e. tiles forming irregular and
> potentially discontiguous coverage of the earth).

Not yet, as far as I know.

We have a proposal in to NASA to develop CF metadata conventions for
satellites and derived products, with a university partner and
scientists who work with satellite data.  If it's funded, I'm sure
we'll be involved with conventions for tiling and swaths.

--Russ

Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: JLN-510178
Department: Support netCDF
Priority: Normal
Status: Closed