[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #MJW-484586]: a few more questions on fletcher 32 checksum

Hi Randy,

> I could not find a lot at the NetCDF website on how to use [the
> Fletcher32 checksum functions]
> I did run across setting some constants ...
> #define
> NC_NOCHECKSUM<http://www.unidata.ucar.edu/software/netcdf/docs_snapshot/html/netcdf_8h.html#ac42e062a3be9d>
> a20b62239f383760b0c> 0
> In HDF5 files you can set check-summing for each
> variable. <http://www.unidata.ucar.edu/software/netcdf/docs_snaps>
> hot/html/netcdf_8h.html#ac42e062a3be9da20b62239> f383760b0c>
> #define
> NC_FLETCHER32<http://www.unidata.ucar.edu/software/netcdf/docs_snapshot/html/netcdf_8h.html#a8598aa723c600>
> cc97e76a5f11ea8abd4> 1
> In HDF5 files you can set check-summing for each
> variable. <http://www.unidata.ucar.edu/software/netcdf/docs_snaps>
> hot/html/netcdf_8h.html#a8598aa723c600cc97e76a5> f11ea8abd4>
> And I did come across a couple of C functions ...
> EXTERNL int nc_def_var_fletcher32 (int ncid, int varid, int fletcher32)
> nc_inq_var_fletcher32<http://www.unidata.ucar.edu/software/netcdf/docs_snapshot/html/group__variables.>
> html#ga2e4999ccf1df31d8aebcc0989913bccc> (int ncid, int varid, int 
> *fletcher32p)
> Learn the checksum settings for a variable.

Those are the netCDF functions you would use.  You shouldn't need to
call any HDF5 functions.  You call nc_def_var_fletcher32 for each
variable that you want checksummed.  You ordinarily wouldn't need to
call nc_inq_var_fletcher32, because once a variable is defined as
checksummed, accessing it always checks the checksums, and writing to
it always updates the checksums, chunk-wise (that is each chunk has a
checksum, and writing to a chunk will cause that chunks checksum to be
updated).  Whether a variable is checksummed or not should be
transparent to readers, except that they will get an error if a chunk
is ever read with an incorrect checksum.

> So it looks like you can cause HDF5 libraries to calculate the
> checksums on writes, and append the checksums to the "chunk" you are writing.
> Is there a means to enable/disable using the checksum on reads ?

No, as far as I understand, if you set checksumming for a variable, it
will always be checked on reads and updated on writes.

> Note that I found the function calls in the HDF5 library at
> http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Proper> ty-SetEdcCheck ...
> Sometimes, it is desirable to skip checksumming during data reading
> in order to get good performance.? There is a 
> function H5Pset_edc_check to offer this option:
> herr_t H5Pset_edc_check(hid_t dxpl, H5P_EDC_t check)

That's never called in the netCDF-4 library implementation, so it's
not available for use.  If there is a real need for it, it might be
possible to add in the future.  Note that the checksum is only checked
when a chunk is read into the chunk cache, so if a reading process
reads data from the same chunk while it's still in the cache, there's
no performance hit on the read.

> Do I have any misunderstandings ?

It sounds like your understanding is right.


Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu

Ticket Details
Ticket ID: MJW-484586
Department: Support netCDF
Priority: Normal
Status: Closed

NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.