Re: [netcdfgroup] [EXTERNAL] Re: Can I build zstandard support in netCDF with a static build?

  • To: Ed Hartnett <edwardjameshartnett@xxxxxxxxx>
  • Subject: Re: [netcdfgroup] [EXTERNAL] Re: Can I build zstandard support in netCDF with a static build?
  • From: "Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC]" <matthew.thompson@xxxxxxxx>
  • Date: Tue, 2 Aug 2022 20:20:17 +0000
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nasa.gov; dmarc=pass action=none header.from=nasa.gov; dkim=pass header.d=nasa.gov; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=GzsH7qTUNNxt4ou7n3zBGEKsictYm6SVgCnRdlw9Vaw=; b=dKkd9pSGfGxnTijZR44FDa3m7XFYG/9DashY04Hdk4mzSSnsk3WewapJZ4UaURDoiBREeNxp5u0Q15NyfPq30a/eqkgrqfUVmP1IHURUYrbmyPcSoLppl61iQsvcK4uqNxKCY/nDBOySJmzKdStfUbOseT12Z+24VjQmowkkYXNICrJwTRqqlpTKGqMWi2KDMuz2PogzQ76q0v2jEc+Kfd6JbiQYCy8MXg2jwrkLyUxBBOpvPkKlZUGg8e/3MFKI74+8vy58ZwdFeC9bwyhisnTukhyoeP1xvrnov1jIR6ZxvmkpAo0tRCEhL0LEC1ukR5CznstQDnY8F8JTO1akZg==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=du8p7i0LwklrgwYSgCrhxLf8ybkgRPnTU8axLKMjt/ZDNkcLCgc6sFe+rPumZbMmsHBDa40t/bbWf3/FtVFDck8aRlt2CQaR4doZf9TSeRxCXRuv1PoYkznxE27UNKATzXyF04a3bW+PCw2hqyNYaGx6oy8QrhD7q3B8tkFYWflYE+j23gNVE15r99ZFnRMFv4hnOS1lFQ9av4EIRAWSJtShtn5ToAQTFnmVY1xJDZgZWt5Pz9pXskPCNyy/LoF9QDX/rCFdhSK7fGBnyY61iPuHQ19s/u/XuqHJEzbK9g4NBxD9jqA7yBqomcpq0PanNpwtBUAPsnhQIlU45u6fLA==
  • Authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nasa.gov;
Ed,

Well, I suppose it was inevitable and I haven't made a massive change in GEOS' 
build system in a while...

I went back to my last attempt to build CCR (someday I'll get it working!), and 
I think I need to add:


  1.  HDF5: --with-default-plugindir=$(HDF5_PLUGIN_PATH)
  2.  netCDF-C: --with-plugin-dir=$(HDF5_PLUGIN_PATH)
  3.  NCO: --with-hdf5-plugin-path=$(HDF5_PLUGIN_PATH)

I suppose I should ask Charlie Zender about the last one...but it seems right.

Now for some reason I can't get the "make check" step for netCDF-fortran to 
recognize HDF5_PLUGIN_PATH, so the zstandard tests fail...but I can run them by 
hand. Not sure if it's on my end or I'm not passing things down right to the 
check step? :shrug:

Now onto testing it. Be prepared for questions like "What are good values to 
set zstandard_level and quantizing and ...". (Well, first I'll go read your 
papers and the tests in the code, but then I'll probably annoy you :D )

Matt

--
Matt Thompson, SSAI, Ld Scientific Programmer/Analyst
NASA GSFC,    Global Modeling and Assimilation Office
Code 610.1,  8800 Greenbelt Rd,  Greenbelt,  MD 20771
Phone: 301-614-6712                 Fax: 301-614-6246
http://science.gsfc.nasa.gov/sed/bio/matthew.thompson

From: Ed Hartnett <edwardjameshartnett@xxxxxxxxx>
Date: Tuesday, August 2, 2022 at 1:03 PM
To: "Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC]" 
<matthew.thompson@xxxxxxxx>
Cc: "netcdfgroup@xxxxxxxxxxxxxxxx" <netcdfgroup@xxxxxxxxxxxxxxxx>
Subject: [EXTERNAL] Re: [netcdfgroup] Can I build zstandard support in netCDF 
with a static build?

Howdy Matt!

For zstandard, and other filter features, you must build and use shared 
libraries.

Also you need to build netcdf-c with --with-plugins=/some/path and then set 
environment variable HDF5_PLUGIN_PATH to /some/path before you build 
netcdf-fortran (or use any program with the zstandard filter).

Ed

On Tue, Aug 2, 2022 at 10:57 AM Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND 
APPLICATIONS INC] via netcdfgroup 
<netcdfgroup@xxxxxxxxxxxxxxxx<mailto:netcdfgroup@xxxxxxxxxxxxxxxx>> wrote:
All,

I recently grabbed netCDF-C v4.9.0 and netCDF-Fortran v4.6.0 and thought, heck, 
let's try to get zstd support as well! zstd built just fine and when I built 
netCDF-C it seemed pretty happy:

  ZSTD Support:           yes

but then when I tried to build netCDF-Fortran:

  configure: WARNING: ------------------------------------------
  configure: WARNING: libnetcdf was built with zstd support, but 
HDF5_PLUGIN_PATH is not set.
  Either set the HDF5_PLUGIN_PATH environmental variable if zstandard tests 
fail,
  or use --disable-zstandard-plugin when running configure.
  configure: WARNING: ------------------------------------------

Okay. New to me. I then went back to the netCDF-C build and saw:

  checking whether dynamically loaded plugins is enabled... yes
  configure: WARNING: --disable-shared => --disable-plugins
  ...
  checking whether and where we should install plugins... no
  ...
  Plugin Install Prefix:  N.A.

Seeing that warning reminded me I build netCDF-C and netCDF-Fortran as a static 
library mainly for historical reasons (was static in the past and, well, if it 
works, don't mess with it unless you need to!)

So I suppose the question is: Can I use zstd with a static build of netCDF?

Or is it time to bite the bullet and reconfigure my software to use shared 
libraries...

Matt

--
Matt Thompson, SSAI, Ld Scientific Programmer/Analyst
NASA GSFC,    Global Modeling and Assimilation Office
Code 610.1,  8800 Greenbelt Rd,  Greenbelt,  MD 20771
Phone: 301-614-6712                 Fax: 301-614-6246
http://science.gsfc.nasa.gov/sed/bio/matthew.thompson
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web.  Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.


netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx<mailto:netcdfgroup@xxxxxxxxxxxxxxxx>
For list information or to unsubscribe,  visit: 
https://www.unidata.ucar.edu/mailing_lists/<https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.unidata.ucar.edu%2Fmailing_lists%2F&data=05%7C01%7Cmatthew.thompson%40nasa.gov%7Cc26dbcf169054901b36308da74a8fb15%7C7005d45845be48ae8140d43da96dd17b%7C0%7C0%7C637950566361758136%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=vLPKypJxoyONwV0DqbPbTFlxXTt7Ao9n0Ff7QBAAliU%3D&reserved=0>
  • 2022 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdfgroup archives: