[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

NPP files

Dave Santek wrote:

John,
Any update to the 4 issues you list below? Tommy is beginning to look at the NPP VIIRS data files. My first goal is to be able to deal with the union aggregation of these files so we can associate the data with the geolocation [since they are in separate files]....issue #2 below. Even being able to use NcML would be a great start!
Do you have any thoughts on how we can get from Point A to Point B? We have time & people to throw at this problem. 

Thanks,
dave

John Caron wrote:

Hi Dave:
My apologies for losing your email down the cracks. 4.1 is really now the recommended version, we just havent had a chance to stabilize the code, but all the changes are in the point data stuff.
Im reviewing our emails to make sure i understand what you need, which i think is access to NPOESS/NPP swath data through the CDM, to be visualized by McIDAS-V. I assume McIDAS-V == IDV for these purposes? 

The issues that I know about:
1. The NPP files ive seen use HDF5 "region references". Currently our HDF5 IOSP doesnt handle these (one of the few things we dont yet handle in HDF5 format). Region references appear to be a way to specify logical subsets of other data arrays which are in the file. Those "regular" data arrays are accessible through the CDM, so im unclear if we are missing any functionality by not having the region references working. So one task is to understand what the point of the NPOES region references is, and if we really need them. . 
this one is in your court.
2. The CDM needs geolocation. Another task is to charactorize which NPOESS/NPP files have this in the data files, and which have them seperate, and so need a union aggregation.
sounds like the coordinates are in a seperate file. can you count on that? do you know the name?
3. The swath data type is different from grid, particularly in how time dimesnion works. Sometimes the current geogrid implementation works and sometimes not. A task is to upgrade CDM to handle swaths correctly and generally. 4. Aggregation of granules. Current aggregation techniques probably wont work. Solutions depends on the context: a few files or large collections? There may have to be some external indexing for fast searching. First task is to characterize the collections, and the use cases. to
im currrently at NCDC where theyve just agreed to have someone here work on a swath feature type. also they will propose a CF convention. im not sure how fast those will happen. do you want to be involved?
aggregation may be a more difficult matter if you are trying to answer queries like "give me all data in lat/lon and time bounding box", but simpler if you just want to allow index space subsetting. what do you need?
I would see SSEC acting as domain experts in understanding NPOESS/NPP. Unidata will do core CDM work, with SSEC adding whatever specialized things are needed. 


Dave Santek wrote:

Hi John,
I don't think I heard back from you regarding a few questions I had below. If you could supply a few brief answers, it would help in a meeting I'm having this afternoon. 

Thanks,
dave

Dave Santek wrote:

John,

I tried your vsn 4.1 with this file:
GMTCO_npp_d20030125_t084705_e084830_b00015_c20071212222807_den_OPS_SEG.h5
in McIDAS-V and it works great! Do you expect 4.1 to be a stable version soon?
The next step, from my perspective, is getting the above geolocation file associated [union aggregation?] with the data file: VSSTO_npp_d20030125_t084705_e084830_b00015_c20071212222807_den_OPS_SEG.h5
These files do have xml files along with them [which I've attached, for what it's worth].
Any thoughts on how to associate these 2 files with NcML? Is that an issue with your netCDF library or is it at the McIDAS-V/IDV or VisAD layer?
I need a simple plan and approach that can be presented to folks here to get some $$s committed for Tommy's time to begin to look into this. 

Thanks for any advice,
dave


John Caron wrote:

Dave Santek wrote:


John,
As far as I know, the NPOESS format is still a moving target. But, I think it is safe to assume that each band and product will be in a separate file [which include the geolocation being in a separate file]. 

>From the 2008 AMS conference:
http://npoess.noaa.gov/IPOarchive/ED/Outreach_Materials/public_presentations/AMS/2008/Posters/54.NPOESS_Sym_2008_Ullman_R3.pdf
Not only that, but these files will contain only a few minutes of data. They will have to be concatenated to create a useful image for the user. 

In regards to some of your specific questions & statements:
also, aggregating swath data may not be possible, it seems likely that a different method has to be developed.
Is this in reference to concatenating several files for display? 
yes
I agree, that a different method will need to be developed. Or, is this about getting the geolocation aggregated with the image data? 
not sure, but may be able to use existing "union" aggregation.
Im guessing we want to write a CoordSysBuilder class to recognize NPP/NPOESS data. This will obviate the NcML. I will get a start on that, and then hopefully you can continue it and perhaps keep improving it as we better understand the data.
Do we need to make this NPOESS specific right away? Is there still work to be done just on the generic reading of HDF5 and aggregating with separate geolocation files, that can be prototyped with NcML? 
possibly - lets get some example files and see.
BTW, latest release of 4.1 has an NPP CoordSysBuilder in it, which works, eg on
ftp://ftp.ssec.wisc.edu/pub/incoming/GMTCO_npp_d20030125_t084705_e084830_b00015_c20071212222807_den_OPS_SEG.h5 

though not on
ftp://ftp.ssec.wisc.edu/pub/incoming/VSSTO_npp_d20030125_t084705_e084830_b00015_c20071212222807_den_OPS_SEG.h5 


(since theres no geolocation in that file)
what i need from you guys at first is domain expertise, to learn what npp is doing and if its documented, if we have the latest file formats, what their intention is on adding geolocation inside the files, etc. i know at some point they were planning to keep various information in external files. is that still the case? so theres a lot of work to be done understanding how these files are intended to be used, and figure out if we can fix the problems in the CoordSysBuilder.
what are your thoughts? are there other issues that i am missing?
I certainly don't claim to be the expert, but I am in possession of a DVD containing a lot of sample data!
If it makes sense, I would like to concentrate on bettering the netCDF-4 interface to HDF-5 so that IDV, McIDAS-V are able to display the data, even if it means creating a NcML file at first. Does that make sense?
yes, and, creating an NcML file can be as daunting as having to program in java.
One complaint we hear from researchers is that is takes a Java programmer to incorporate new data. If the data are stored in a standard format, HDF-5 for example, it shouldn't take a programmer in order to get a quick look at the data.
yes, but if the geolocation is missing, how do you know what it should be ?? if there is some standard way of knowing that, then we can hopefully automate it.
HDFView can easily read & display read a file; my initial goal is to have that basic functionality in IDV/McV with the added geolocation. Even if it means selecting the geolocation as a separate step or through NcML. 

Thanks,
dave

John Caron wrote:

Dave Santek wrote:

John,
As part of the AMS abstract I sent in last month, was the statement: "The use of Unidata’s network Common Data Form (netCDF) version 4 along with netCDF Markup Language (NcML) files to more easily handle a variety of Hierarchical Data Format (HDF) version 4 and 5 files. This is being applied to simulated imager and sounder data for the upcoming National Polar-orbiting Operational Environmental Satellite System (NPOESS) mission."
We've had a few email exchanges earlier this year in regards to dealing with 'groups' and 'aggregating' with HDF-5 files. Has much changed recently in the netCDF 4 library along these lines?
I'd like to get Tommy Jasmin working on this within the next few weeks. Can you offer some guidance on where to start and what needs attention? 

Thanks,
dave

Hi Dave, Tommy:

Sorry for the delay in getting back to this.
I am trying to catch back up on this; reviewing your support email started 12/22, has these 2 files:
"I have 2 HDF5 files I'd like to visualize in IDV/McIDAS-V. One of the files is the geolocation [along with TerrainHeight, zenith angles, etc.]: ftp://ftp.ssec.wisc.edu/pub/incoming/GMTCO_npp_d20030125_t084705_e084830_b00015_c20071212222807_den_OPS_SEG.h5 

The other contains SSTs:
ftp://ftp.ssec.wisc.edu/pub/incoming/VSSTO_npp_d20030125_t084705_e084830_b00015_c20071212222807_den_OPS_SEG.h5"; <ftp://ftp.ssec.wisc.edu/pub/incoming/VSSTO_npp_d20030125_t084705_e084830_b00015_c20071212222807_den_OPS_SEG.h5> 

The first is amenable to ncml, as it has the lat/lon arrays:

<?xml version="1.0" encoding="UTF-8"?>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2"; location="GMTCO_npp_d20030125_t084705_e084830_b00015_c20071212222807_den_OPS_SEG.h5"> 

<attribute name="Conventions" value="CF-1.0" />

<group name="All_Data">
<group name="VIIRS-MOD-GEO-TC_All">
<dimension name="scan" length="3200" />
<dimension name="xscan" length="768" />

<variable name="Latitude" shape="xscan scan" type="float">
<attribute name="units" value="degrees_north" />
</variable>
<variable name="Longitude" shape="xscan scan" type="float">
<attribute name="units" value="degrees_east" />
</variable>

<variable name="TerrainHeight" shape="xscan scan" type="float">
<attribute name="coordinates" value="Latitude Longitude" />
</variable>
</group>
</group>

</netcdf>
i think groups is only an issue when the coordinates are in different groups than the data variables, which doesnt happen here.
But ftp://ftp.ssec.wisc.edu/pub/incoming/VSSTO_npp_d20030125_t084705_e084830_b00015_c20071212222807_den_OPS_SEG.h5"; <ftp://ftp.ssec.wisc.edu/pub/incoming/VSSTO_npp_d20030125_t084705_e084830_b00015_c20071212222807_den_OPS_SEG.h5> has no lat/lon geolocation in the file.
also, aggregating swath data may not be possible, it seems likely that a different method has to be developed.
however, we can start looking at npp data seriously and see how far we can get.
Im guessing we want to write a CoordSysBuilder class to recognize NPP/NPOESS data. This will obviate the NcML. I will get a start on that, and then hopefully you can continue it and perhaps keep improving it as we better understand the data.
what i need from you guys at first is domain expertise, to learn what npp is doing and if its documented, if we have the latest file formats, what their intention is on adding geolocation inside the files, etc. i know at some point they were planning to keep various information in external files. is that still the case? so theres a lot of work to be done understanding how these files are intended to be used, and figure out if we can fix the problems in the CoordSysBuilder. 

what are your thoughts? are there other issues that i am missing?