[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: netCDF question....

On Dec 18,  4:44pm, Leigh Angus wrote:
> Subject: netCDF question....
> #
> #                                            18 December 1997
> #
>
> Hello Russ:
>
> My name is Leigh Angus and I am a meteorologist/programmer with CIRA
> at the NOAA Forecast System Lab here in Boulder.
> I sent you a question via email in the past and appreciated your quick and
> helpful reply.  I have a quick question about netCDF access that I would
> like to run by you.

In general, it is a better idea to email 'address@hidden' than
to email us directly. That way the question is "in the system" and you will
be assured of an answer even if say Russ in his closet writing a paper.

> First a bit of an overview of what we are trying to do.
> Our group over here is working on the Local Data Acquisition and
> Dissemination (LDAD) feature for AWIPS.  The LDAD system basically
> has the responsibility of bringing into a weather forecast office all
> of the local surface weather data sets that might exist in a particular
state.
> These data are then decoded and stored into netCDF data files.  In
> addition to storing these data into netCDF files, we have to
> display the weather data on our D2D (AWIPS) workstation.  Our
> dillema is that it takes too long to read the data from the netCDF
> data files in time to display the data on the workstation screen.
>
> We are storing data from a bunch of weather stations in these netCDF
> data files.  There might be, for example,  5000 stations in a netCDF data
> file each reporting 10 or so pre-determined weather parameters
> (e.g., temperature, wind speed, etc..). So for each station in the
> netCDF file there is a value for the weather station name, its temperature,
> its wind speed, etc.  The UNLIMITED dimension is the number of records in
> the file, which is not known ahead of time.

Just to be sure that I understand you, each station report corresponds
to a record, right?

> So, my question for you is as follows.  Is there a way that we can
> read in a netCDF file in a piece of (C/C++ code, for example) in one
> "fread()" call and then parse out the individual fields from the
> netCDF file by some sort of byte/offset method?

You probably don't want to do this unless other options have been exhausted.
Get familiar with the performance section of the user's guide:
http://www.unidata.ucar.edu/packages/netcdf/guidec/guidec-14.html#HEADING14-0

Some general things you can do to improve performance:
        1) Use netcdf-3.
        2) Be sure the file is on local disk as opposed to NFS.
                (If you must access the file over NFS, let me know, there are
                some other tips I can give you.)
        3) Take advantage of the netcdf libary buffering by avoiding use
                of the NC_SHARE option when opening the file, OR
        3) Use the contributed mmapio i/o layer to memory map the file.
                (You probably can't mmap an NFS file.)
        4) Access the variables of a "record" sequentially.
                (If you need help understanding what I mean by this,
                send me the output of 'ncdump -h' of one of your typical
                data sets, and I'll get specific.)

It is important to profile your program and see where the problem really lies.
I can't tell you how many times we've recieved questions like this only to
discover (upon analysis) that the problem lay elsewhere. A common problem
in a situation like yours is that the reader does repeated linear searches
through the file to find records (stations) of interest.

If you are concerned about multiple calls for the various variables making up
your record, you can save some overhead by using the netcdf-2 'ncrecget()'
calls. It has been our experience that using this call instead of (properly
ordered) sequential calls to nc_get_varxxx doesn't save very much time.

> We would also have to know how to read past the netCDF file header as well by
> knowing how much space the header takes up before the actual data values
> are stored.

If you really want to try this, The file format is documented in the
User's Guide in Appendix B.
http://www.unidata.ucar.edu/packages/netcdf/guidec/guidec-18.html#HEADING18-0

>
> We need FAST netCDF data access......
> I would appreciate any information you could provide on this matter.
> Thanks a lot.
>
> Leigh Angus                                     address@hidden
> CIRA                                            phone: (303) 497-6187
> NOAA/FSL   R/E/FS6                              fax:   (303) 497-7256
> 325 Broadway
> Boulder, CO   Room 269.