Hi Scott,
> As part of my research, I have been producing large numbers of large
> NetCDF files (10-30GB) using WRF-Chem. The size of the files is starting
> to become a problem, as I am running out of space both on my local hard
> drive and on my allocation of the remote computer I use. It also makes
> transfer of files between computers very time consuming. Much of this
> space use is unavoidable. However, wen using WRF-Chem, any variables
> which are not used by the current schemes are still outputted into the
> NetCDf file, with each data point given a value of 0. Thus, the final
> file ends up containing huge 4-dimentional arrays containing nothing
> but zeros but still using as much space as the useful data.
>
> Do you know if there are any programs that exist which can read through
> a NetCDF file, deleting any variable which only contains zero values? Or
> if not, could you offer me any advise on how to write a script which
> could perform this task (my preferred languages are c++ and Fortran95).
No, sorry, I don't know of any programs that can do that. The netCDF
API doesn't support deleting a variable from an existing file. To get
files without the all-zero variables would require writing a program
that copies the no-zero variables from an existing file to a new file,
detecting the variables that are all zero and not copying them.
The current netCDF-beta release includes an nccopy utility that copies
a netCDF file with various options for specifying the output format
variant or for compressing all the variables in the output copy, which
would then be a netCDF-4 classic model file. The netCDF-3 format
doesn't support per-variable compression, but netCDF-4 format and
netCDF-4 classic model format can use compression, which would be very
effective on variables that are all zero. If the programs that read
your files are linked against the netCDF-4 library, they can
transparently uncompress data on the fly, as the data is read, using
chunking, so only the data that is read is uncompressed, rather than
the whole file.
So here are three possible approaches:
1. Write a pogram, based on modifying nccopy, that would copy only
variables that have non-zero values, and use it to convert your
files to new files that omit such variables. This approach may
be costly in computer time, becasue you would have to read all
your existing datasets and write new ones without the all-zero
variables, in addition to the effort required to modify the
nccopy program to do justt what you want.
2. Use the netCDF-4 classic model format supported by the netCDF-4
library whenever you write new files, and specify use of
compression (deflation) when you create new variables, as
documented in the netCDF-4 Users Guide. Then all new files will
be much smaller even with variables containing nothing but
zeros. But programs that read the new data files will have to
be linked against tthe netCDF-4 library, which can also access
data in the old netCDF-3 classic format files.
3. Use the nccopy utility and its -d option to deflate (compress)
variables in the output copies, making the copies much smaller.
Again, the programs that later read the data will have to be
linked against the ntCDF-4 libraries.
--Russ
Russ Rew UCAR Unidata Program
address@hidden http://www.unidata.ucar.edu
Ticket Details
===================
Ticket ID: OYD-297799
Department: Support netCDF
Priority: Normal
Status: Closed
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.