Due to the current gap in continued funding from the U.S. National Science Foundation (NSF), the NSF Unidata Program Center has temporarily paused most operations. See NSF Unidata Pause in Most Operations for details.
Due to the current gap in continued funding from the U.S. National Science Foundation (NSF), the NSF Unidata Program Center has temporarily paused most operations. See NSF Unidata Pause in Most Operations for details.
On Apr 14, 2014, at 6:31 PM, Kent Yang <myang6@xxxxxxxxxxxx> wrote: I don’t think your programs cause this. For unlimited dimensions, the HDF5 chunking storage is used. There are space overheads for chunks, especially when the chunk size is small. Exactly -- very small chunk sizes can make for huge files. But this seems like a lot of overhead to me. Did you use the defaults for chunk sizes? I would have thought they'd do pretty well for this case, but it's certainly worth trying setting them by hand to see what you get. Some googling will find you some good advice, but this is a good start: http://www.unidata.ucar.edu/blogs/developer/en/entry/chunking_data_why_it_matters Chris One way to mitigate this issue is to increase the chunk size and then to use compression. Kent *From:* netcdfgroup-bounces@xxxxxxxxxxxxxxxx [mailto: netcdfgroup-bounces@xxxxxxxxxxxxxxxx] *On Behalf Of *Wei Huang *Sent:* Monday, April 14, 2014 4:38 PM *To:* NetCDF Mailing List *Subject:* [netcdfgroup] NetCDF-4 filesize question. Hello group, I am doing some NetCDF-4 test, and find when I use fixed, and unlimited dimensions cause file size changed dramatically. Below is a list of files (with dimension 5 x 10 x 73 x 144, with few group names): 1. has one unlimited dimension (the most left one) 2. has two unlimited dimensions (the left two) 3. has fixed dimensions. -rw-r--r-- 1 huangwei CIT\Domain Users 6312828 Apr 14 15:25 NCLcreatedNC4.nc.1unlimited -rw-r--r-- 1 huangwei CIT\Domain Users 10508612 Apr 14 15:24 NCLcreatedNC4.nc.2unlimited -rw-r--r-- 1 huangwei CIT\Domain Users 2112758 Apr 14 15:26 NCLcreatedNC4.nc.fixed My question is: does our program (has issues which) cause the file size difference, or, is it NetCDF4 which needs the extra space (for unlimited dimension)? Thanks, Wei Huang _______________________________________________ netcdfgroup mailing list netcdfgroup@xxxxxxxxxxxxxxxx For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/
netcdfgroup archives: