[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #KLB-596506]: apparent bug in netcdf-4.2

Wei-keng,

> Here is what I found. When entering nc_redef and exiting with nc_enddef,
> NC_begin in nc.c is called and it recalculates the file starting
> offsets for all variables, regardless the offsets in the existing file
> (i.e. ncp->begin_var and ncp->vars.value[*]->begin). Is it possible that
> the "gaps" between variables due to PnetCDF's 512-byte alignment are
> mistaken as part of the variables when the header size grows and all
> variables must be "shifted" down? In file nc.c, line 604, the assertion
>                 assert(gnu_off > old_off);
> check if the new begin of a variable is larger then the old one.
> I wonder why the assertion did not fail.

By the way, in the current 4.3-rc2 version and the upcoming 4.3.0
release, the functions that used to be in libsrc/nc.c are now in
libsrc/nc3internal.c, due to more refactoring of the dispatch layer.

I don't think the gaps between variables are being added to the header
size.  Whether or note the header size has changed, it must still be
rewritten on nc_endef, because netCDF objects may have just been
renamed, perhaps with names of the same or smaller lengths than before.
I don't think the assert(gnu_off > old_off) is getting called, because
move_vars_r() and move_recs_r() aren't getting called.

> It is interesting that the CDF-1 format has no such problem, but
> PnetCDF does the same alignment for CDF-1 as well. Did you
> use nccopy to convert the CDF-2 to CDF-1 and run the test? In that
> case, I guess all alignments have been removed during the copy.

Exactly right.

> Now the question is whether netCDF would like to support this non-record
> variable alignment that PnetCDF is using. If not, PnetCDF can simply
> remove this feature to be compatible with netCDF (performance might
> suffers in some cases, though). If yes, what are the good and bad
> for other netCDF tools and backward compatibility?

We want to support it!  Once this library bug is fixed, I think there
won't be any bad side-effects, except the one you have already noted,
that nccopy doesn't preserve the efficient alignment that pnetcdf uses.
I should note that in the nccopy documentation.

If we wanted nccopy to preserve your alignment, we would need a new
function to detect and make that available to nccopy and other
third-party tools.  Maybe ncdump -s should display it as one of the
"special" performance-related variable attributes, such as _Format and
all the netCDF-4 chunking, compression, and similar special attributes.

> One more thing, when you mentioned
> > the library has made the additional assumption of 4-byte alignment
> > within the fixed data section at least since version 3.6.0 in 2004
> > (I've tested that and intervening versions have the same bug).
> 
> Do you mean "within the the fixed data section" that each fixed variable's
> begin must aligned with 4 bytes? Is netCDF doing this alignment? In file
> nc.c lines 209 and 210
>                 (*vpp)->begin = index;
>                 index += (*vpp)->len;
> It does not appear to me the alignment is done.

I'll have to check that, I though variable offsets, record boundaries,
and most metadata objects in the header were always 4-byte aligned.  But
I see calls with v_align and r_align set to 1 rather than 4, so I may be
wrong about that.

> PnetCDF's default alignment is 512 bytes which is already a multiple
> of 4, and fulfills the requirement of 4-byte alignment.
> I checked the file offsets of all variables in test_orig.nc and they
> all seem to be aligned with 512 byte boundaries. Did you find otherwise?

Right, that was done by pnetcdf, but an nccopy of the file doesn't have
512 byte alignment.  Whether it has 4-byte alignment or not, I'll have
to check.

> Am I missing something here?

Nope, I think you understand the essence of the problem.  I just have to
get some dedicated time to fix it and test my fix.

Thanks again for your assistance!

Regards,
--Russ

Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: KLB-596506
Department: Support netCDF
Priority: Normal
Status: Closed