[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: netcdf ncsync() bug

> Subject: Re: netcdf ncsync() bug
>
> Hello,
>
> A while ago I reported a problem I was having with ncsync() in 2.3.2.  I've
> attached a reply from Russ which summarizes the problem.  In some recent
> testing with netCDF 2.4-beta the problem still appears, and a quick check
> of the code looks like the only change was the addition of some comments.
> Was this discovered to not be a bug? or not worth fixing?  I was hoping the
> fix would appear in 2.4, but perhaps the email just got lost in the volumes
> which you must receive.  If nothing else, I want to cast a vote for the
> continued consideration of real-time uses of netcdf.
>
> Thanks,
> Gary
>
> ----
> Gary Granger                            address@hidden
> Research Data Program                   http://www.atd.ucar.edu/rdp/gjg.html
> National Center for Atmospheric Research  Ph: 303-497-8816  FAX: 303-497-8770

Gary:

Thanks for reminding us about this.

The sample fix you sent us seems to indicate that you have multiple
programs with the same netcdf open for writing, and you want to synchronize
the header with calls to ncsync().

I've modified that fix slightly to avoid some work in some cases.
I include that at the end of this message.

I have no problem in principle of nc_sync() reading the header even for
read/write files. However, I must caution you:
        ncnobuf() is a noop at this point, all operations are buffered
in xdrposix.c.
        The NC_NSYNC and NC_HSYNC mechanisms are an undocumented hack.
(I think they were added in response to requests from Jon Corbett.) E.G.,
we don't test them properly.
        It is quite easy fool oneself into thinking multiprocessing is
working, when in fact one is just lucky while testing.
        The layout of the netcdf file is not particularly condusive to
efficiency of in ncsync().

In summary, "multiple concurrent writers" is not a supported feature
of netcdf 2.x. The previous definition of ncsync() is simply a manifestation
of that fact. The original logic says, I've got it open read/write,
so no one else should (multiple writers unsupported), so, I don't need to read.

The good news is that all this is changing.
I'm working on netcdf 3.0, which includes a complete reimplementation of
netcdf/libsrc while retaining the current file format. Some specific
goals of the reimplementation are support of locking and process level
concurrency.

I'll let you know when it's ready.

-glenn


ncsync(id)
int id ;
{
        NC *handle ;
        int reRead = 1; /* flag to read the header */

        cdf_routine_name = "ncsync" ;

        handle = NC_check_id(id) ;
        if(handle == NULL)
                return(-1) ;

        if( handle->flags & NC_INDEF )
        {
                NCadvise(NC_EINDEFINE, "Unfinished definition") ;
                return(-1) ;
        }

        if(handle->flags & NC_RDWR)
        {
                handle->xdrs->x_op = XDR_ENCODE ;
                if(handle->flags & NC_HDIRTY)
                {
                        if(!xdr_cdf(handle->xdrs, &handle) )
                                /* MEMORY: should xdr_free(xdr_cdf, &handle) be
                                 * called? */
                                return(-1) ;
                        handle->flags &= ~(NC_NDIRTY | NC_HDIRTY) ;
                        reRead = 0;
                }
                else if(handle->flags & NC_NDIRTY)
                {
                        if(!xdr_numrecs(handle->xdrs, handle) )
                                /* MEMORY: should xdr_free(xdr_numrecs,
                                 * handle) be called? */
                                return(-1) ;
                        handle->flags &= ~(NC_NDIRTY) ;
                }
        }

        (void) NCxdrfile_sync(handle->xdrs) ;

        if(reRead != 0)
        {
                /* free the stuff in handle that xdr_cdf allocates */
                handle->xdrs->x_op = XDR_FREE ;
                (void) xdr_cdf(handle->xdrs, &handle) ;

                /* reread the header */
                handle->xdrs->x_op = XDR_DECODE ;
                if(!xdr_cdf(handle->xdrs, &handle) )
                {
                        nc_serror("xdr_cdf") ;
                        /* MEMORY: should xdr_free(xdr_cdf,
                         * &handle) be called? */
                        NC_free_cdf(handle) ; /* ?? what should we do now? */
                        return(-1) ;
                }
                if( NC_computeshapes(handle) == -1)
                        return(-1) ;
        }

        return(0) ;
}