Use nccopy to convert, rechunk, and compress the data. How long does reorganizing 17.5 GB of data take?

• Initially, nccopy took too long (over 24 hours!), because it used default chunk cache of only 4 MB. For comparison, file took 5.5 minutes to write, 4.75 minutes to read and check, 15 minutes to just copy to netCDF-4 classic.
• HDF5 utility h5repack takes even longer than nccopy, uses only 1 MB for chunk cache and has no options to specify more.
• Added nccopy options for specifying chunk cache (in snapshot, not 4.1.3 release). Now nccopy rechunks data in about 14 minutes.
• Results speed up time-series access significantly, but also slows accessing lat/lon data.
• Rechunking to just favor accessing data for a specific location, using 1698 x 25 x 24 chunks, results in very fast access by location (0.1 sec instead of 10.3 sec) but very slow access by time (230 sec instead of 0.25 sec).
• Using default chunking (100 x 96 x 94) and compressing

        nccopy -d1 cvx3.nc cvx4-d1.nc
    

  took about 15 minutes to copy and compress, resulting in 6.1 GB file. Accessing data for a specific location is fairly fast (1.2 sec instead of 10.3 sec), but getting all data for a specific time is now slower (13.2 sec instead of 0.25 sec).
• Work remains to have nccopy figure out how much chunk cache it needs and allocate it automatically when rechunking.
• This is complicated because chunk shapes on input and output may be very different, no way to optimize both reading and writing access while copying and rechunking.

2011 Unidata NetCDF Workshop > Example of the Importance of the Chunk Cache