Richard,
I'm not sure if you *need* to, but maybe it is more convenient. From my
testing, I think if you build NCO with that flag, then you don't need any
environment variables, but I could get it to work with an older NCO with
HDF5_PLUGIN_PATH set in the environment. (If Charlie is on, he can edify us
both).
So, I have an old NCO 5.0.7 that I built with netCDF's that didn't know about
zstandard at all.
And if I try some ncks on it:
❯ ncks --version
NCO netCDF Operators version 5.0.7 "Mariupol" built by mathomp4 on
gs6101-parcel.ndc.nasa.gov at Jul 27 2022 07:35:52
ncks version 5.0.7
~/ZstandardTest
at 10:56:21 AM
❯ echo $HDF5_PLUGIN_PATH
❯ ncks case1_zstd.2004.nc4
netcdf case1_zstd.2004 {
dimensions:
...
data:
ERROR: nco_get_vara() failed to nc_get_vara() variable "VAR2D"
nco_err_exit(): ERROR Short NCO-generated message (usually name of function
that triggered error): nco_get_vara()
nco_err_exit(): ERROR Error code is -136. Translation into English with
nc_strerror(-136) is "NetCDF: Filter error: unimplemented filter encountered"
nco_err_exit(): ERROR NCO will now exit with system call exit(EXIT_FAILURE)
Okay then, but if I set my HDF5_PLUGIN_PATH to the plugins I just built:
❯ export
HDF5_PLUGIN_PATH=/Users/mathomp4/Baselibs/ESMA-Baselibs-7.6.0-TEST/x86_64-apple-darwin21.6.0/gfortran/Darwin/lib/hdf5_plugins
❯ ncks case1_zstd.2004.nc4
...
data:
VAR2D =
-167.125, -167.125, -167.125, -167.125, -167.125, -167.125, -167.125,
-167.125, -167.125, -167.125, -167.125, -167.125, -167.125, -167.125, -167.125,
-167.125, -167.125, -167.125, -167.125, -167.125, -167.125, -167.125, -167.125,
-167.125, -167.125, -167.125, -167.125,
Data!
Meanwhile, I built NCO 5.1.0 with --with-hdf5-plugin-path and I can I read the
file just fine whether or not HDF5_PLUGIN_PATH is set!
I might have belt-and-suspendered my builds. Maybe as long as the underlying
HDF5 knows about the HDF5_PLUGIN_PATH environment variable, many utilities will
work?
Matt
--
Matt Thompson, SSAI, Ld Scientific Programmer/Analyst
NASA GSFC, Global Modeling and Assimilation Office
Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771
Phone: 301-614-6712 Fax: 301-614-6246
http://science.gsfc.nasa.gov/sed/bio/matthew.thompson
From: netcdfgroup <netcdfgroup-bounces@xxxxxxxxxxxxxxxx> on behalf of Richard
van Hees <R.M.van.Hees@xxxxxxx>
Date: Wednesday, August 3, 2022 at 1:55 AM
To: "netcdfgroup@xxxxxxxxxxxxxxxx" <netcdfgroup@xxxxxxxxxxxxxxxx>
Subject: Re: [netcdfgroup] [EXTERNAL] Re: Can I build zstandard support in
netCDF with a static build?
Thanks, I had not realized that I needed te recompile NCO as well.
To my understanding, the need to define HDF5_PLUGIN_PATH is not required when
all the plugins are in the default plugin directory, right?
A little annoyance: I always compile HDF5 without szip support, but now with
v4.9.0 the 'make check' of netCDF4 fails because I do not have szip support.
Thanks for all the good work.
Regards, Richard van Hees
On 02/08/2022 22:20, Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND
APPLICATIONS INC] via netcdfgroup wrote:
[EXTERNAL EMAIL]
Ed,
Well, I suppose it was inevitable and I haven't made a massive change in GEOS'
build system in a while...
I went back to my last attempt to build CCR (someday I'll get it working!), and
I think I need to add:
1. HDF5: --with-default-plugindir=$(HDF5_PLUGIN_PATH)
2. netCDF-C: --with-plugin-dir=$(HDF5_PLUGIN_PATH)
3. NCO: --with-hdf5-plugin-path=$(HDF5_PLUGIN_PATH)
I suppose I should ask Charlie Zender about the last one...but it seems right.
Now for some reason I can't get the "make check" step for netCDF-fortran to
recognize HDF5_PLUGIN_PATH, so the zstandard tests fail...but I can run them by
hand. Not sure if it's on my end or I'm not passing things down right to the
check step? :shrug:
Now onto testing it. Be prepared for questions like "What are good values to
set zstandard_level and quantizing and ...". (Well, first I'll go read your
papers and the tests in the code, but then I'll probably annoy you :D )
Matt
--
Matt Thompson, SSAI, Ld Scientific Programmer/Analyst
NASA GSFC, Global Modeling and Assimilation Office
Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771
Phone: 301-614-6712 Fax: 301-614-6246
http://science.gsfc.nasa.gov/sed/bio/matthew.thompson
From: Ed Hartnett
<edwardjameshartnett@xxxxxxxxx><mailto:edwardjameshartnett@xxxxxxxxx>
Date: Tuesday, August 2, 2022 at 1:03 PM
To: "Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC]"
<matthew.thompson@xxxxxxxx><mailto:matthew.thompson@xxxxxxxx>
Cc: "netcdfgroup@xxxxxxxxxxxxxxxx"<mailto:netcdfgroup@xxxxxxxxxxxxxxxx>
<netcdfgroup@xxxxxxxxxxxxxxxx><mailto:netcdfgroup@xxxxxxxxxxxxxxxx>
Subject: [EXTERNAL] Re: [netcdfgroup] Can I build zstandard support in netCDF
with a static build?
Howdy Matt!
For zstandard, and other filter features, you must build and use shared
libraries.
Also you need to build netcdf-c with --with-plugins=/some/path and then set
environment variable HDF5_PLUGIN_PATH to /some/path before you build
netcdf-fortran (or use any program with the zstandard filter).
Ed
On Tue, Aug 2, 2022 at 10:57 AM Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND
APPLICATIONS INC] via netcdfgroup
<netcdfgroup@xxxxxxxxxxxxxxxx<mailto:netcdfgroup@xxxxxxxxxxxxxxxx>> wrote:
All,
I recently grabbed netCDF-C v4.9.0 and netCDF-Fortran v4.6.0 and thought, heck,
let's try to get zstd support as well! zstd built just fine and when I built
netCDF-C it seemed pretty happy:
ZSTD Support: yes
but then when I tried to build netCDF-Fortran:
configure: WARNING: ------------------------------------------
configure: WARNING: libnetcdf was built with zstd support, but
HDF5_PLUGIN_PATH is not set.
Either set the HDF5_PLUGIN_PATH environmental variable if zstandard tests
fail,
or use --disable-zstandard-plugin when running configure.
configure: WARNING: ------------------------------------------
Okay. New to me. I then went back to the netCDF-C build and saw:
checking whether dynamically loaded plugins is enabled... yes
configure: WARNING: --disable-shared => --disable-plugins
...
checking whether and where we should install plugins... no
...
Plugin Install Prefix: N.A.
Seeing that warning reminded me I build netCDF-C and netCDF-Fortran as a static
library mainly for historical reasons (was static in the past and, well, if it
works, don't mess with it unless you need to!)
So I suppose the question is: Can I use zstd with a static build of netCDF?
Or is it time to bite the bullet and reconfigure my software to use shared
libraries...
Matt
--
Matt Thompson, SSAI, Ld Scientific Programmer/Analyst
NASA GSFC, Global Modeling and Assimilation Office
Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771
Phone: 301-614-6712 Fax: 301-614-6246
http://science.gsfc.nasa.gov/sed/bio/matthew.thompson
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web. Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx<mailto:netcdfgroup@xxxxxxxxxxxxxxxx>
For list information or to unsubscribe, visit:
https://www.unidata.ucar.edu/mailing_lists/<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.unidata.ucar.edu%2Fmailing_lists%2F&data=05%7C01%7Cmatthew.thompson%40nasa.gov%7Ce1b4c7bb93c547dde0a808da7514d112%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637951029522576734%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2FPSkk6y985KKUfwOtQnnXUSQ%2Fi%2F3TvwYmGUaluIAEyU%3D&reserved=0>
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web. Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx<mailto:netcdfgroup@xxxxxxxxxxxxxxxx>
For list information or to unsubscribe, visit:
https://www.unidata.ucar.edu/mailing_lists/<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.unidata.ucar.edu%2Fmailing_lists%2F&data=05%7C01%7Cmatthew.thompson%40nasa.gov%7Ce1b4c7bb93c547dde0a808da7514d112%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637951029522576734%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2FPSkk6y985KKUfwOtQnnXUSQ%2Fi%2F3TvwYmGUaluIAEyU%3D&reserved=0>
