[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: suggestions for formatting "station data" into NetCDF (fwd)

>To: address@hidden
>From: Dennis Shea <address@hidden>
>Subject: Re: 20041119: suggestions for formatting "station data" into NetCDF 
>(fwd)
>Organization: NCAR and NOAA National Ocean Service
>Keywords:

Hi Dennis,

> I have often thought about the issue outlined below.
> A while ago I tried to search for netCDF-stations data files.
> I was not very successful.
> 
> netCDF for, say, rawin observations at standard levels would
> be easy, something like (there are many other permutations)...
> 
>    DENVER (time,number_variables,level)
> 
> Thing is, in the not so ideal "real world", stations
> report irregularly, they may have "significant levels"
> at each time, blah-blah.
> 
> I am just not sure what the most robust way of doing this.
> Do you have any suggestions or pointers to other efforts.
> 
> THX
> D

Yes, John Caron has worked out a structure for station data in netCDF
that has several advantages.  Here is an example CDL of the data,
which uses CF conventions vocabulary but also makes use of some
additional conventions not covered by CF to represent station data:

  http://my.unidata.ucar.edu/content/software/netcdf/examples/metar.cdl

NOAA's MADIS data uses a similar structure.  Here is an example of
that, for a single hour's worth of data:

  http://my.unidata.ucar.edu/content/software/netcdf/examples/madis-metar.cdl

and similarly for SAO data

  http://my.unidata.ucar.edu/content/software/netcdf/examples/madis-sao.cdl

These are designed to make some patterns of access more efficient.
For example, if you want all the data for a particular station, you
can use the prevReport variable to quickly access the previous report
from the same station.

--Russ


> ---------- Forwarded message ----------
> Date: Fri, 19 Nov 2004 09:38:24 -0500
> From: Mark Vincent <address@hidden>
> To: address@hidden
> Cc: address@hidden, Greg Mott <address@hidden>,
>      tom gross <address@hidden>, Mark Vincent <address@hidden>
> Subject: suggestions for formatting "station data" into NetCDF
> 
> Hi Dennis,
> 
> Zack contacted you recently about decoding BUFR files to NetCDF.  One of
> our researchers had some old NWS fortran code that has successfully done
> the decoding part and we have temporaly written it out to ascii to see
> what the data looks like.  Unforntately, the data is not as well posed
> are we usually deal with.
> 
> In general specs. of the BUFR files (and the temporary ascii file) are:
> 
> - produced once per hour
> - contain about 200 stations
> - potentially the number of stations may vary
> - about 12 variable types (not all stations have all variables)
> - the stations don't report at all the same time (i.e. may each station
> may have a diff. time per file at minutes 00, 17, 02 etc.)
> - most stations report once per hour but .....
> - some stations report more than once per hour
> - we are going to use this data to produce model data sets
> - our developer wants to preserve all the data (i.e the ones with
> multiple records per hour) and not loose any time resolution (i.e. "bin"
> them to even incremets of say 00,15,30,45)
> 
> For example a quick example may look like:
> 
> record number
> station
> date
> time
> wind
> clouds
> etc. etc. etc
> etc...12
> 1
> 1
> 20041119
> 0813
> 15
> 15
> 15
> 15
> 15
> 15
> 2
> 2
> 20041119 0800
> -9999 15
> -9999 15
> 15
> -9999
> 3
> 3
> 20041119
> 0800
> 15
> 15
> -9999
> -9999
> -9999
> -9999
> 4
> 3
> 20041119
> 0812
> 15
> `15
> 15
> 15 15
> 15
> 5
> 3
> 20041119
> 0824
> 15
> 15
> 15
> 15
> 15
> 15
> 6
> 3
> 20041119
> 0836
> 15
> 15
> 15
> 15
> 15
> 15
> 7
> 3
> 20041119
> 0847
> 15
> 15
> 15
> 15
> 15
> 15
> 8
> 4
> 20041119
> 0801
> 15
> -9999
> -9999
> -9999
> -9999
> -9999
> .........
> 5
> 20041119
> 0837
> -9999
> -9999
> -9999
> 15
> 15
> 15
> .........
> 6
> 20041119
> 0812
> -9999
> 15
> 15
> 15
> 15
> 15
> 200
> 7
> 20041119
> 0847
> 15
> -9999
> 15
> -9999
> 15
> -9999
> 
> 
> 
> Can you give us a quick suggestion on how to dimension this data. We can
> do the coding, but are having an in-house fundamental diagreement on how
> to dimension the data.
> 
> For example the two polar ideas are:
> option 1) the data (wind for example) will be a function of time and
> station. Allow for up to 60 records per hour (i.e. once per minute) and
> place the data in the appropriate minute of data and fill the rest with
> missing values.  This would preserve the data but would require a 60
> fold level of padding for stations that only report once per hour.
> 
> option 2) make the data (and times) a function of record number only
> 
> other options ) ?????
> 
> We would appreicate your recommendation and any guidance on how to avoid
> "criminal" netcdf programming.
> 
> Thanks!
> 
> Mark
> 
> -- 
> Mark S. Vincent, Ph.D.
> NOAA National Ocean Service
> Center for Operational Oceanographic Products and Services (CO-OPS)
> SSMC4 - 7307  N/OPS3
> 1305 East-West Highway
> Silver Spring, MD 20910
> 
> phone: 301-713-2890 ext 151
> internet: address@hidden