[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Query re copying of netCDF files

Hi Harvey,

> Can you please clarify the conditions under which the whole file is
> copied (to a tmp copy)? Section 9.1 of User Guide seems to imply that
> this happens whenever ncendef is called.  Is this correct (in both
> versions 2.4 & 3)?

In section 9.1, the sentence

    A potential disadvantage of this organization is that any operation
    on a netCDF file that requires expanding the header, for example
    adding new dimensions and new variables to an existing netCDF file,
    will be as expensive as copying the file.  This expense is incurred
    when @code{ncendef()} is called, after a call to @code{ncredef()}.

implies to me that the file should not be copied if you are only
changing existing attribute values, without changing their type to
something that requires more space.  That is my understanding of how
things should work.  

> The issue arose from users of FAN here & at GFDL who are accessing
> 500MB NCEP files with wrong valid_range attribute values (defined as
> scaled rather than unscaled values!)  I advised them to use text2nc to
> correct these values & assumed (since header does not increase in
> size) it would be done in place.  But they both reported that this was
> not the case.

To correct existing attribute values, it should not be necessary to call
ncredef() or ncendef().  If you just ncopen() the file, call ncattput()
to change existing attribute values to different values of the same
type, and call ncclose(), no copy takes place.  I've just tested this
with netcdf 2.4 to make sure the i-node of the file was the same before
and after the new attribute values were written.  A CDL and test program
are appended if you want to verify this works.

I tried the same test program, except that I inserted an ncredef() call
before the ncattput() call and an ncendef() call after.  Although these
calls were unnecessary and the ncattput was only changing existing
attribute values, this resulted in an unnecessary copy of the file.  I
think this is a bug.

> I am in the process of converting FAN to use netCDF 3.0.  Will this
> solve the problem?

I haven't tried this on netCDF 3.1a yet.

--Russ

-------------- f.cdl:
netcdf f {
dimensions:
        n = 3;
variables:
        float fp(n) ;
              fp:valid_range = -5.f, 5.f ;
data:
    fp =  1, 2, -3 ;
}
-------------- ncendef-test.c:
#include "netcdf.h"

int
main() {                        /* see if ncattput/ncendef copies file */

   int  ncid;                   /* netCDF id */
   int  fp_id;                  /* variable id */
   float  fp_valid_range[2];    /* attribute vector */

   ncid = ncopen("f.nc", NC_WRITE);
   fp_id = ncvarid(ncid, "fp");
   fp_valid_range[0] = -4;
   fp_valid_range[1] = 4;
   ncattput (ncid, fp_id, "valid_range", NC_FLOAT, 2, (void *) fp_valid_range);
   ncclose (ncid);
   return 0;
}