[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #KLB-596506]: apparent bug in netcdf-4.2

Jim,

> It looks like that file was originally created on bluefire on 11/21/11, I
> don't have any information about which netcdf library was used, but I think
> that some adjustment may have been made inside netcdf for performance on
> gpfs filesystems.
> 
> But doesn't your own
> 
> int nc__enddef(int ncid, size_t h_minfree, size_t v_align,
> size_t v_minfree, size_t r_align);
> 
> 
> allow for changing this alignment?   I don't know that that was done for
> this file, but it would seem to suggest that there is no assumption being
> violated about these alignments.  Or that one part of netcdf is assuming
> something which another part is not.

That function, and in particular the v_align argument, is only supposed to 
control the alignment of the *beginning* of the data section for fixed size 
variables, not the alignment of each variable in that data section.  It's
possible that either the documentation is incorrect or the library doesn't 
implement the documented behavior correctly.

In any case, you may be right that two parts of the library interpreting this
alignment specification differently may be the root cause of the problem.

When I figure it out, I'll let you know if we can fix it ...

--Russ

> address@hidden> wrote:
> 
> > Hi Jim,
> >
> > I'm curious how the original file you provided was created and perhaps
> > modified.  It has a peculiar alignment characteristic that I haven't
> > seen before, and if there are more netCDF files being created the same
> > way, we may nned to adapt.
> >
> > Could you tell me the history of the file, what file system it was
> > written on, and whether the netCDF library with which it was written
> > was modified in any way?
> >
> > The file has this characteristic, which would indicate a non-Posix
> > file system: it is using 512-byte alignment of data values rather than
> > the 4-byte alignment assumed by netCDF. So, for example, the data
> > block for fixed-size variables begins with 9 scalar integers that
> > should take 4 bytes each. The offsets computed for these values from
> > the beginning of the fixed-size data block are 0, 4, 8, 12, 16, 20,
> > 24, 28, 32, so there is no padding or wasted space. The offsets from
> > the beginning of the fixed-size data block that are actually stored in the
> > header for these variables are 0, 512, 1024, ... , 4096. If the file
> > system used to write the data originally could not write data on
> > 4-byte boundaries, I think that violates the assumption of netCDF and
> > POSIX I/O. Nevertheless, if the nc_endef() call pays attention to the
> > file offsets for each variable that are stored in the header (as the
> > netCDF library does when reading the file), rather than computing them
> > from assuming 4-byte alignment, perhaps this file can be modified
> > correctly.
> >
> > The function where we might be able to adapt to this is
> > nc3internal.c:NC_begins(), which is called from
> > nc3internal.c:NC_enddef().  In any case it's a netCDF bug to write
> > something that can't be later read correctly, so if our unmodified
> > library wrote that file and we can't adapt to it, then it was a bug
> > to not emit an error message for trying to create a file on the original
> > non-POSIX file system.  Also, the data seems to all be there in the
> > "corrupted" file, which can be fixed by just restoring the variable
> > offsets in the file header to the peculiar values in the original ...
> >
> > --Russ
> >
> > Russ Rew                                         UCAR Unidata Program
> > address@hidden                      http://www.unidata.ucar.edu
> >
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: KLB-596506
> > Department: Support netCDF
> > Priority: Normal
> > Status: Closed
> >
> >
> 
> 
> --
> Jim Edwards
> 
> CESM Software Engineering Group
> National Center for Atmospheric Research
> Boulder, CO
> 303-497-1842
> 
> 
Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: KLB-596506
Department: Support netCDF
Priority: Normal
Status: Closed