[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #DTM-721292]: netcdf file size discrepancy across machines

Hi Marc,

> I am finding a peculiar instance that when an identical netcdf file
> is written via NCL (NetCDF 4 Classic, compression level 5,) on one
> machine the file is 5.6GB but on the other computer its size is 9GB.
> To the best of my knowledge, the files are identical when opened (type,
> values, etc.)  What is odder is that if I execute the NCL script (and
> ensure they are using the same version of NCL) from the command line
> instead of from the shell script, the file sizes are identical.
> 
> Are there certain checks I am missing that I should try?
> 
> I believe both machines are using the same NetCDF libraries, but could
> that be the issue?

Yes, that might explain what you are seeing.  If you are not specifying
how to chunk variables that are compressed, then default chunking is used.
The algorithm for deciding what chunk sizes to use as a default has changed
several times in different versions of the netCDF-4 library, as we discovered
inefficiencies in default chunk sizes.

One way to check this is with the "-s" and "-h" options of ncdump, assuming
you have recent enough versions of ncdump to support the "-s" option.  That
will show you the chunk sizes along each dimension for each variable.  If
those turn out to be different, that could explain the huge difference in
compression.  If the chunk sizes are the same, I'm stumped.  In the latter
case, I would forward the question to NCL support, they may have an
explanation (which I would like to hear about also).

--Russ

> Anyhow, I am somewhat stumped and believe it must be a machine environment
> issue difference when both machines execute the shell script that calls
> the NCL script to generate the netcdf.
> 
> Any help you could lend would be greatly appreciated...thanks!
> 
> -Marc
> 
> 

Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: DTM-721292
Department: Support netCDF
Priority: High
Status: Closed