[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: continuous archiving with HDF

Claude Saunders wrote:
> 
> We currently use an in-house file format for all our accelerator
> data and a suite of analysis tools. I would like to use HDF as
> a replacement for the underlying storage engine. One obstacle
> is the fact that presently we read and plot files that are
> currently open and being actively appended by some tool. This
> is handled by file locking and well defined flushing of buffers
> to the file, so there is no chance of reading a partially written
> data record or header.
> 
> Does anyone do this with HDF? It doesn't seem to support this
> level of control (and quite possibly can't be reasonably expected
> to).
> 
> --
> /--------------------------------------------------------------------\
>  | Claude Saunders              | Operations Analysis Group         |
>  | Argonne National Laboratory  | address@hidden                  |
> \--------------------------------------------------------------------/

I know that the netCDF interface supports a single writer and
multiple readers simultaneously. The specific details depend
depend on which version of netCDF one is using. In the version 2
interface, there were calls 'ncnobuf()' and 'ncsync()'. In version
3, one opens the file with a flag NC_SHARE. We plan to add locking
in netCDF 3.2.

I don't know if the "netCDF interface on top of HDF files"
implementation supports this or not.

-glenn