[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #IJX-817619]: HDF4 chunking (and compression?) support hints

Hi Charlie,

I'm just jumping in on top of Russ's last email.  

It seems like we might have an easier short-term solution.  Currently the 
information for the version number and the capabilities of the library (since 
many capabilities like dap, netcdf-4, etc. are optional) can be accessed two 
ways:

1. By parsing the output of `nc-config --all` (or even parse the 'nc-config' 
file directly).  You can alternatively query nc-config for specific things, 
e.g. 
   $ nc-config --version
   $ nc-config --has-nc4

etc.  

2. Starting with the 4.3.3 release candidate, netcdf creates a file 
'libnetcdf.settings' which is installed alongside the netcdf library.  This 
file can also be parsed, and contains additional information above 'nc-config' 
such as the 'configured on' date.

Out of curiosity, is this determination of netcdf capabilities needed at 
configure time or runtime?  If the former, and if you are using autotools for 
your build system, you can have it query the netcdf library for the presence of 
specific functions.  For example, after locating the hdf5 library with 
AC_SEARCH_LIBS, we use the following stanzas in our configure.ac to determine 
whether or not libhdf5 was built with parallel IO support:

   AC_CHECK_FUNCS([H5Pget_fapl_mpiposix H5Pget_fapl_mpio H5Pset_deflate 
H5Z_SZIP])

   # The user may have parallel HDF5 based on MPI POSIX.
   if test "x$ac_cv_func_H5Pget_fapl_mpiposix" = xyes; then
      AC_DEFINE([USE_PARALLEL_POSIX], [1], [if true, compile in parallel 
netCDF-4 based on MPI/POSIX])
   fi

   # The user may have parallel HDF5 based on MPI mumble mumble.
   if test "x$ac_cv_func_H5Pget_fapl_mpio" = xyes; then
      AC_DEFINE([USE_PARALLEL_MPIO], [1], [if true, compile in parallel 
netCDF-4 based on MPI/IO])
   fi

There is also a way to incorporate a shell script in configure.ac which could 
be used to parse libnetcdf.settings for the version.  I was sure I had done 
this (to mirror functionality in our cmake-based build) to enforce a minimum 
libhdf5 version but I see now that it's not in there.  So, I will need to redo 
this.

The problem with adding NC_HAVE_* in netcdf.h is that it would turn netcdf.h 
into a generated file instead of a static file, which could become problematic 
as it would just grow and grow over time as we add additional functionality.  
Plus, it would duplicate symbols already tracked in the internal 
non-distributed 'config.h' file, which would case problems on the development 
side.

Would parsing the extant nc-config or libnetcdf.settings work?

-Ward




> Hi Charlie,
> 
> > Now that NCF-272 is fixed, I would like to support it in NCO.
> > What is the best way? To workaround the now-solved bug,
> > NCO currently avoids all chunking/compression calls on HDF4 files.
> > Chunking should now be allowed on HDF4 files as long as
> > NCO was built with netCDF 4.3.3+.
> > So...we are in the familiar yet uncomfortable position where
> > NCO needs to implement version-dependent libnetcdf.a functionality.
> >
> > For whatever reason, netCDF does not include a YYYYMMDD date token
> > (like GCC does) nor a library version token (also like GCC) with
> > which clients like NCO could implement netCDF version/date specific
> > workarounds using those as pre-processor conditionals.
> > In the absence of version/date tokens, the next easiest way for us is if
> > netcdf.h includes a function-specific indicator token, e.g.,
> > NC_HAVE_HDF4_CHUNKING
> > NC_HAVE_HDF4_DEFLATE
> 
> Would macros such as the__GNUC__, __GNUC_MINOR__, __GNUC_PATCHLEVEL__
> described at
> 
> https://gcc.gnu.org/onlinedocs/cpp/Common-Predefined-Macros.html
> 
> suffice for what NCO needs (substituting "NETCDFC" for "GNUC" in the
> macro names), or do you have a better suggestion?  It seems that just
> having a version string wouldn't work very well, becasue youy want o
> know whwther the current version is later than a specific version, and
> the approach with the macros above allows this.  The alternative of a
> version date integer such as YYYYMMDD would require developers to know
> about the correspondence between version numbers and release dates, so
> I'd prefer not to use that.
> 
> > Also, it is unclear if netCDF-HDF4 now supports compression
> > inquiry and/or setting functions. When I filed the original
> > bug report nc_inq_var_deflate() would crash on HDF4. Does it still?
> 
> Not sure, we'll have to get back to you on that.
> 
> > It would be great if such library hints could be in 4.3.3-final.
> 
> If it's just a matter of defining them in netcdf.h for this and all
> subsequent releases, we can do that.  We'd define them manually
> for this release and automate it as part of the release management
> from now on.
> 
> --Russ
> 
> > Thanks,
> > Charlie
> > --
> > Charlie Zender, Earth System Sci. & Computer Sci.
> > University of California, Irvine 949-891-2429 )'(
> >
> >
> 
> Russ Rew                                         UCAR Unidata Program
> address@hidden                      http://www.unidata.ucar.edu
> 
> 


Ticket Details
===================
Ticket ID: IJX-817619
Department: Support netCDF
Priority: Normal
Status: Closed