Hi Mark, > This issue is closely related to Support Ticket #CUV-251255 and Jira > issues NCF-85 and NCF-73. > > I am trying to rechunk a large (43GB) NetCDF 4 file into a form where > the chunks are optimsed for the read patterns that I am using. The file > structure is > > dimensions: > lat = 720 ; > lon = 1440 ; > date = 5186 ; > variables: > int date(date) ; > date:_Storage = "chunked" ; > date:_ChunkSizes = 1 ; > date:_Endianness = "little" ; > float lat(lat) ; > lat:long_name = "latitude" ; > lat:units = "degrees_north" ; > lat:axis = "Y" ; > lat:_Storage = "contiguous" ; > float lon(lon) ; > lon:long_name = "longitude" ; > lon:units = "degrees_east" ; > lon:axis = "X" ; > lon:_Storage = "contiguous" ; > float CHL1_mean(date, lat, lon) ; > CHL1_mean:long_name = "Chlorophyll-a concentration. Case 1 water - Mean of > the binned pixels" ; > CHL1_mean:_FillValue = -999.f ; > CHL1_mean:units = "mg/m3" ; > CHL1_mean:pct_characterised_error = 53.04f ; > CHL1_mean:_Storage = "chunked" ; > CHL1_mean:_ChunkSizes = 1, 720, 1440 ; > > I would like to rechunk the variables so that instead of being stored as > date/1,lat/720/lon1440 that they are stored as date/1,lat/30,lon/30. I > am using a version of nccopy compiled from the nightly builds with the > following command: > > nccopy -m 1G -h 2G -e 10000 -c lon/30,lat/30 step1.nc step2.nc > > This command, however hangs - it writes about 20GB of the 40GB to step2.nc > within the first 5 minutes, but after that the file size remains static > for 13 hours (even though nccopy is still active and occupying RAM). At > this point I loose patience. > > However, I believe I have found the cause of the error. I was able to > read the header of the (broken) 20GB file produced, and it showed that > the chunks that were being created were in fact date/5186,lon/30,lat/30 > rather than the date/1,lon/30,lat/30 that I was aiming for. I would guess > that the (unintended) change in the date chunking is coming from nccopy. > > So, the questions are: > > 1. why does nccopy hang when trying to create chunks that are > 5186 x 30 x 30 ? I'm not sure, and would need to duplicate that behavior here to see what's happening. Is it possible to supply me with a test file or a small program that generates a test file that demonstrates the problem? > 2. is it desired behaviour that nccopy changes the chunking of dimensions > that are not-specified in the re-chunking command? No, I think that's a bug. Does it work when you explicitly specify "-c date/1,lon/30,lat/30" ? --Russ Russ Rew UCAR Unidata Program address@hidden http://www.unidata.ucar.edu Ticket Details =================== Ticket ID: AWT-862217 Department: Support netCDF Priority: Normal Status: Closed
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.