[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: NPP files



Tom Rink wrote:
All,

john caron wrote:
Dave Santek wrote:
John,

Any update to the 4 issues you list below? Tommy is beginning to look at the NPP VIIRS data files. My first goal is to be able to deal with the union aggregation of these files so we can associate the data with the geolocation [since they are in separate files]....issue #2 below. Even being able to use NcML would be a great start!

Do you have any thoughts on how we can get from Point A to Point B? We have time & people to throw at this problem.

Thanks,
dave

John Caron wrote:
Hi Dave:

My apologies for losing your email down the cracks. 4.1 is really now the recommended version, we just havent had a chance to stabilize the code, but all the changes are in the point data stuff.

Im reviewing our emails to make sure i understand what you need, which i think is access to NPOESS/NPP swath data through the CDM, to be visualized by McIDAS-V. I assume McIDAS-V == IDV for these purposes?

The issues that I know about:

1. The NPP files ive seen use HDF5 "region references". Currently our HDF5 IOSP doesnt handle these (one of the few things we dont yet handle in HDF5 format). Region references appear to be a way to specify logical subsets of other data arrays which are in the file. Those "regular" data arrays are accessible through the CDM, so im unclear if we are missing any functionality by not having the region references working. So one task is to understand what the point of the NPOES region references is, and if we really need them. .
this one is in your court.

2. The CDM needs geolocation. Another task is to charactorize which NPOESS/NPP files have this in the data files, and which have them seperate, and so need a union aggregation.
sounds like the coordinates are in a seperate file. can you count on that? do you know the name?
Do you mean the geolocation coordinates? If so, I think we can count on them
being located in a different file.
is the file name derivable from the original name? where does that file come from?

3. The swath data type is different from grid, particularly in how time dimesnion works. Sometimes the current geogrid implementation works and sometimes not. A task is to upgrade CDM to handle swaths correctly and generally.

This seems a question of whether an Eulerian or Lagrangian view of measurement
is more appropriate, with regard the time dimension.

4. Aggregation of granules. Current aggregation techniques probably wont work. Solutions depends on the context: a few files or large collections? There may have to be some external indexing for fast searching. First task is to characterize the collections, and the use cases. to
im currrently at NCDC where theyve just agreed to have someone here work on a swath feature type. also they will propose a CF convention. im not sure how fast those will happen. do you want to be involved?

We certainly should be involved in that, and should have some input. In fact, we already have an effective swath feature type in McIDAS-V which handles multi and hyper-spectral data like AIRS, MODIS, AVHRR, IASI wherein instrument-specific peculiarities and
requirements are handled.
this would be swath feature type within the CDM, analogous to GridDatatype. Perhaps you should extract your public API as a starting point?


aggregation may be a more difficult matter if you are trying to answer queries like "give me all data in lat/lon and time bounding box", but simpler if you just want to allow index space subsetting. what do you need?

A big step forward would be able to union files together so we can
treat them as one, for example, of the same time range.  We should
be able to handle the relationship between lon/lat/time and index
space.

that should already work. have you tried it?