[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 20020816: netCDF question

>To: <address@hidden>
>From: "Chris Wells" <address@hidden>
>Subject: netCDF question
>Organization: ?
>Keywords: 200208161322.g7GDMbK14622 netCDF structures

Hi Chris,

>       I'm considering the use of netCDF for the storage of
> satellite imagery. I've reviewed all the documents on your site;
> here's a question I couldn't glean an answer to.
> 
>       I need to store scans of data which are packaged as
> C struct's, similar to the following:
> 
> struct Scan
> {
>       double SecondsOfDay ;
>       struct Position Nadir ;
>       char Pixels[1465] ;
>       int QualityFlags[1465] ;
> };
> 
> struct Position
> {
>       double latitude ;
>       double longitude ;
>       double altitude ;
> } ;
> 
> struct Scan Scans[43000] ;
> 
> I'd like the Scan structure to be my fundamental unit of I/O.
> Is there a way to do such a thing in netCDF, and if so, how?

No, but you can do something similar.  If you really need arrays of
structs as the fundamental unit of I/O, you probably want HDF5 instead
of netCDF:

  http://hdf.ncsa.uiuc.edu/HDF5/

The closest you can get with netCDF is flatter than what you want,
because netCDF doesn't support nested data structures (because of its
need to support access from Fortran-77 as well as C).  Here's one
version of the corresponding netCDF structure:

 netcdf Scans {

 dimensions:
   nScans = unlimited ;
   nPixels = 1465;

 variables:
   double SecondsOfDay(nScans) ;
   double latitude(nScans) ;
   double longitude(nScans) ;
   double altitude(nScans) ;
   byte Pixels(nScans, nPixels) ;
   int QualityFlags(nScans, nPixels) ;
 }

This would store each scan contiguously in a netCDF "record", with the
complete set of 43000 scans in 43000 records, the size of the
unlimited dimension.  It would support fast random access to any
specified scan and permit efficient appending of new scans.  There
would be very little wasted space, just 3 extra bytes of padding per
record, to make the int array start on a 4-byte boundary.  

However the I/O for accessing a scan would not be done as a single
operation, though you could make it appear this way with your own
writeScan(int scanNumber, struct Scan scan) and readScan(...)
functions that call the necessary netCDF functions.  To read a scan
record, for example, would actually require four calls to read the
four doubles, one call to read the byte array of Pixels, and one call
to read the byte array of QualityFlags.

The netCDF-2 C interface included the functions ncrecput() and
ncrecget() to wrap the necessary calls for you, and these are still
supported in the current netCDF library:

  http://www.unidata.ucar.edu/packages/netcdf/guide_toc.html#SEC63
  http://www.unidata.ucar.edu/packages/netcdf/guide_toc.html#SEC67

As stated in the User's Guide appendix for netCDF-2 transition:

  The new C interface omits three "record I/O" functions, ncrecput,
  ncrecget, and ncrecinq, from the netCDF-2 interface, although these
  functions are still supported via the netCDF-2 compatibility
  interface.

  This means you may have to replace one record-oriented call with
  multiple type-specific calls, one for each record variable. For
  example, a single call to ncrecput can always be replaced by
  multiple calls to the appropriate nc_put_var functions, one call for
  each variable accessed. The record-oriented functions were omitted,
  because there is no simple way to provide type-safety and automatic
  type conversion for such an interface.

--Russ

_____________________________________________________________________

Russ Rew                                         UCAR Unidata Program
address@hidden                     http://www.unidata.ucar.edu