Hi Dave, > I am one of the group that provides support to the WRF model. One of the > most used data formats for the WRF model is NetCDF, largely because of the > huge selection of post-processing tools that easily allows diagnostics and > visualization. The WRF model is used in production at NOAA, and as such the > operations staff are always quite sensitive to the amount of time any model > takes. A simple method to reduce the wall-clock time is taken seriously. > > The NOAA developers have asked a couple of questions about the FILL / NOFILL > option (here we reference this UNIDATA page): > http://www.unidata.ucar.edu/software/netcdf/docs/netcdf-c/nc_005fset_005ffill.html > > In our model history volumes, if we only open the files for write, then the > NOFILL option seems to be (not only) an OK choice, but perhaps one that > provides a timing reduction. Our tests indicate that a write with the NOFILL > option is a 1x write, compared to a 2x write + 1x read when doing the FILL > option. Would you please verify this for us? We think that we would like to > set the NOFILL option as the default option in WRF, to benefit our entire > community. Would you please comment on our possible plans in this regard to > help us know if this is the right move. All of our data has TIME as the > unlimited dimension. Other than the HDF5 compression capabilities from > NETCDF4, we are a pretty vanilla user group of NetCDF. I agree that setting the NOFILL option will save time in writing your model history volumes, and would be a reasonable default. The issue is whether the performance benefit outweighs any problems that might occur in determining what data was written and what data are essentially garbage values in case you write the history data variables in a different order used to define the variables in creating the file. For example, if you write the last record variable first and the writing is interrupted, then the values of the previous record variables in that record could be arbitrary. But if you log the creation of history volumes so you know when the writing is complete and buffers are flushed, either by a close or sync call, that seems like it shouldn't be a problem. One other issue is a potentially serious bug related to use of nofill mode and a specific pattern of writing data in versions of netCDF before version 4.1.3 (released in June 2011): [netcdfgroup] Important: potential file corruption using NOFILL mode http://www.unidata.ucar.edu/mailing_lists/archives/netcdfgroup/2011/msg00137.html If you're sure that WRF users use version 4.1.3 or greater of the netCDF C library, or that WRF doesn't meet the conditions described above that trigger the bug, then I think it's safe. There was also a less serious bug with use of fill mode in netCDF versions before 4.2, released in May 2012: Fixed turning off fill values in HDF5 layers when NOFILL mode is set in netCDF-4 API https://bugtracking.unidata.ucar.edu/browse/NCF-151 > Secondly, we are interested in knowing about the statement: > The use of this feature may not be available (or even needed) in future > releases. Programmers are cautioned against heavy reliance upon this feature. > If we get timing reductions, we would like to rely on the continuing > availability of this feature. I think that statement was overly cautious, because we didn't know whether we could support nofill mode in future versions. Now we're committed to backwards compatibility for such features, and I don't anticipate that support for nofill mode would be dropped. The HDF5 software layer used by netCDF-4 also supports fill values and > Thanks in advance for your time and explanations. You're welcome, I hope this helps. --Russ > Dave > > > David Gill > Software Engineer > address@hidden > office: (303) 497 8162 > fax: (303) 497-8171 > Mailing address: > 3450 Mitchell Lane > Boulder, CO 80301 > > > > Russ Rew UCAR Unidata Program address@hidden http://www.unidata.ucar.edu Ticket Details =================== Ticket ID: IJQ-160428 Department: Support netCDF Priority: Normal Status: Closed
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.