Dear all,
I'm a developer of a fortran code which uses netcdf for I/O
In one of my runs I created a file with some huge array of characters.
The header of the file is the following:
netcdf ndb.BS_COMPRESS0 {
dimensions:
BS_K_linearized1 = 2025000000 ;
BS_K_linearized2 = 781887360 ;
variables:
char BSE_RESONANT_COMPRESSED1_DONE(BS_K_linearized1) ;
char BSE_RESONANT_COMPRESSED2_DONE(BS_K_linearized1) ;
char BSE_RESONANT_COMPRESSED3_DONE(BS_K_linearized2) ;
}
The variable is declared as nf90_char which, according to the
documentation should be 1 byte per element.
Thus I would expect the total size of the file to be 1
byte*(2*2025000000+781887360) ~ 4.5 GB
Instead the file size is 16059445323 bytes ~ 14.96 GB, i.e. 10.46 GB
more and a factor 3.33 bigger
This happens consistently if I consider the file
netcdf ndb {
dimensions:
complex = 2 ;
BS_K_linearized1 = 2025000000 ;
BS_K_linearized2 = 781887360 ;
variables:
float BSE_RESONANT_LINEARIZED1(BS_K_linearized1, complex) ;
char BSE_RESONANT_LINEARIZED1_DONE(BS_K_linearized1) ;
float BSE_RESONANT_LINEARIZED2(BS_K_linearized1, complex) ;
char BSE_RESONANT_LINEARIZED2_DONE(BS_K_linearized1) ;
float BSE_RESONANT_LINEARIZED3(BS_K_linearized2, complex) ;
char BSE_RESONANT_LINEARIZED3_DONE(BS_K_linearized2) ;
}
The float component should weight ~36 GB while the char component should
be identical to before, i.e. 4.5 GB for a total of 40.5 GB
The file is instead ~ 50.96 GB, i.e. again a factor 10.46 GB bigger than
expected.
*Why ?**
***
My character variables are something like
"tnnnntnnnntnnnnnnnntnnnnnttnnnnnnnnnnnnnnnnt..."
but the file size is already like that just after the file creation,
i.e. before filling it.
Few info about the library, compiled linking to HDF5 (hdf5-1.8.18), with
parallel IO support:
Name: netcdf
Description: NetCDF Client Library for C
URL: http://www.unidata.ucar.edu/netcdf
Version: 4.4.1.1
Libs: -L${libdir} -lnetcdf -ldl -lm
/nfs/data/bin/Yambo/gcc-8.1.0/openmpi-3.1.0/yambo_ext_libs/gfortran/mpifort/v4/parallel/lib/libhdf5hl_fortran.a
/nfs/data/bin/Yambo/gcc-8.1.0/openmpi-3.1.0/yambo_ext_libs/gfortran/mpifort/v4/parallel/lib/libhdf5_fortran.a
/nfs/data/bin/Yambo/gcc-8.1.0/openmpi-3.1.0/yambo_ext_libs/gfortran/mpifort/v4/parallel/lib/libhdf5_hl.a
/nfs/data/bin/Yambo/gcc-8.1.0/openmpi-3.1.0/yambo_ext_libs/gfortran/mpifort/v4/parallel/lib/libhdf5.a
-lz -lm -ldl -lcurl
Cflags: -I${includedir}
Name: netcdf-fortran
Description: NetCDF Client Library for Fortran
URL: http://www.unidata.ucar.edu/netcdf
Version: 4.4.4
Requires.private: netcdf > 4.1.1
Libs: -L${libdir} -lnetcdff
Libs.private: -L${libdir} -lnetcdff -lnetcdf
Cflags: -I${includedir}
Best,
D.
--
Davide Sangalli, PhD
CNR-ISM, Division of Ultrafast Processes in Materials (FLASHit) and MaX
Centre
Area della Ricerca di Roma 1, 00016 Monterotondo Scalo, Italy
http://www.ism.cnr.it/en/davide-sangalli-cv/
http://www.max-centre.eu/
