Re: [netcdfgroup] questions about netCDf file size.

• To: LB <bravo.loic@xxxxxxxxx>
• Subject: Re: [netcdfgroup] questions about netCDf file size.
• From: John Caron <caron@xxxxxxxxxxxxxxxx>
• Date: Fri, 27 Aug 2010 08:27:52 -0600

you are using netcdf-4 format i think. this has much more complex storage 
objects than what you are used to with netcdf-3 format.

LB wrote:

Hi,

I'm very new to netCDF and I was surprised by the size of files generated by netCDF.

Here is the header of the netCDF file :

ncdump -h cycle.nc <http://cycle.nc>
netcdf cycle {
dimensions:
        Regime = 3 ;

group: Regimes {
  variables:
        float T41(Regime) ;
        float TTR(Regime) ;
        float N1(Regime) ;

  } // group Regimes

group: Cycles {
  dimensions:
        Cycle = 5 ;
        SousCycle = 11 ;
  variables:
        byte Type(Cycle) ;
        short Nb_lignes(Cycle) ;
        short Indice(SousCycle) ;
        short Repeat(SousCycle) ;
        float Duree(SousCycle) ;

  } // group Cycles

group: Simulation {
  dimensions:
        Temps = UNLIMITED ; // (19168 currently)
  variables:
        float Temps(Temps) ;
        byte Regime(Temps) ;
        float T41(Temps) ;
        float e_p1(Temps) ;
        float Temps_Touche ;
  } // group Simulation
}

The generated file is 3.1 Mo and so 50% heavier than I had expected. Here is how I calculated the expected file size :

name Dimension cell size Nb cell variable size group: Regimes { float T41 Regime 32 3 96 float TTR Regime 32 3 96 float N1 Regime 32 3 96 group: Cycles {

byte Type Cycle 8 5 40 short Nb_lignes Cycle 16 5 80 short Indice SousCycle 16 11 176 short Repeat SousCycle 16 11 176 float Duree SousCycle 32 11 352 group: Simulation {

float Temps Temps 32 19168 613376 byte Regime Temps 8 19168 153344 float T41 Temps 32 19168 613376 float e_p1 Temps 32 19168 613376 float Temps_Touche 32 1 32

        
        
                total size      1994616         Bits
                                        soit    1.99    Mo



Is there anything I forgot ?

Besides, I would like to know how to if there is any efficient way to reduce this size:

- I've tried to use deflate_level=1 to 9 in nf90_def_var for the variables in the simulation group but it didn't change the file size. Is there anything else to do to to let netcdf compress data ? gzip seems to compress the resulting file quite well.

- Is there a way to store 16bits float to compress data (with loss) in order to reduce the size of data where we have a limited resolution ?

Best regards,

--
Loic





------------------------------------------------------------------------

_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx

For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/

• Follow-Ups:
  • Re: [netcdfgroup] questions about netCDf file size.
    • From: LB

• References:
  • [netcdfgroup] questions about netCDf file size.
    • From: LB