Lindsay Pender and other netCDF Oceanographers...
I was really happy with the replies I got from my recent post. Lindsay
Pender (pender@xxxxxxxxxxx) brought up a good point when, talking about
using netCDF for underway data, saying...
>... however came up with some conceptual difficulties when I was
>looking at ways to implement it. It may be that your data is different, but in
>our case we have data coming from many different sources, each with different
>sampling rates. Some of our instruments are sampled at 2.5kHz, while others are
>as slow as once a minute. For an underway data storage system using netCDF how
>do you store such data with only one 'unlimited' dimension?
OSU currently can manage it's data by logging 1 minute averages for
all instruments. No one yet has asked for finer resolution from our
common use equipment. CTD, ADCP, and other such higher resolution
systems are managed and logged by their own software and are currently
independent of the new netCDF system. Soon though, I will need to merge
in some finer res. data (5 second GPS and ADCP). Here is my scheme, and
I'm real curious what kinds of alternatives others can suggest.
I'll see if I can describe my idea with a CDL file. It may not be the best
way, but I guess it will work...
<<< BEGIN multi_res.cdl >>>
netcdf multires {
dimensions:
min_max_mean = 3; // store 3 numbers: min, max, mean
ten_hz = 600; // number of 10.0 hZ samples in 1 minute
five_hz = 300; // number of 5.0 hZ samples in 1 minute
twopoint5_hz = 150; // number of 2.5 hZ samples in 1 minute
one_hz = 60; // number of 1.0 hZ samples in 1 minute
five_second = 20; // number of 0.05 hZ samples in 1 minute
time = unlimited; // the "time" dimension
variables:
long time(time); // seconds since some fixed point in time
float gps_lat(time); // gps latitude in sample period
float gps_lon(time); // gps longitude in sample period
short n_sats(time); // number of satellites used in fix
float raw_gps_lat(time, five_second); // raw gps latitude
float raw_gps_lon(time, five_second); // raw gps longitude
float sea_temp(time, min_max_mean); // sea surface temperature
float towed_cdt_temp (time, ten_hz); // raw CTD temperature
float towed_ctd_cond (time, ten_hz); // raw CTD conductivity
}
<<< END multi_res.cdl >>>
The idea is to pick the least common denominator (1 minute data) and
pack anything that's a finer resolution into a new dimension.
I did try this scheme for a towed vehicle logging/display system, but I
found the netCDF time overhead (on a PC) was too high for me to log real
time, raw 24 hZ CTD data. Too many variables to log -- more than the
simple example above. I still used the same idea, but went to a simpler
ASCII file for quick I/O.
Comments???
Tim Holt, OSU Oceanography
holtt@xxxxxxxxxxxx
