[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Question about netcdf

>To: address@hidden
>From: "Bill Cassanova" <address@hidden>
>Subject: Re: 20040512: Question about netcdf
>Organization: weather.com
>Keywords: 200405122045.i4CKjvtK009313

Hi Bill,

> Here's my scenario...There are two processes.  One is called Server
> and one is called client.  Server is responsible for opening a
> netcdf file and writing values to one of the arrays on a continual
> basis.
> 
> Pseudocode might look something like this:
> [ 1 ] Open netcdf file.
> [ 2 ] Do netCDF setup and data element specifications.
> [ 3 ] Forever{
>             Receive data from some outside source
>             Write that data into the elements as they arrive
>       }
> 
> In essence the netCDF file is being used as a database file that is
> continually being updated.
> 
> On the Client side we simply want to open that file and read data
> from it at the same time the server is writing to it.....My question
> is, "is this possible?"

Yes.  For more details, see the section in the User's Guide
"Synchronize an Open NetCDF Dataset to Disk: nc_sync":

  
http://my.unidata.ucar.edu/content/software/netcdf/guidec/guidec-10.html#HEADING10-322

> Let me extend the scenario and say that the client is reading from
> the netCDF on a continual basis.  Will the client always be able to
> read the most up to date values written to the netCDF from the
> server or must the server first close the file before all data is
> commited?

Yes, you can use nc_sync() but the best way to accomplish this is for
both processes to open the file using the NC_SHARE flag (NF_SHARE or
NF90_SHARE for f77 or f90, respectively).  When a writing process
opens a file with the NC_SHARE flag set, writes are flushed to disk
automatically soon after writing, without waiting to fill a buffer.
Similarly, when a reading process opens a file with the NC_SHARE flag,
nc_sync() is automatically called before each read to ensure that the
latest data is read, rather than stale data that might be in a buffer.
The cost for ensuring this kind of consistency is loss of some I/O
efficiency, since you are giving up the advantages of buffering.

Note that this only works for data values and the size of the
unlimited dimension, not for metadata/schema information.  If the
writer tries to change the schema, such as by adding new attributes,
variables, or dimensions, the reader must close and reopen the file to
see the change.

--Russ