[netCDF #IJQ-160428]: NetCDF no-fill option
- Subject: [netCDF #IJQ-160428]: NetCDF no-fill option
- Date: Fri, 14 Feb 2014 06:25:48 -0700
> I am one of the group that provides support to the WRF model. One of the
> most used data formats for the WRF model is NetCDF, largely because of the
> huge selection of post-processing tools that easily allows diagnostics and
> visualization. The WRF model is used in production at NOAA, and as such the
> operations staff are always quite sensitive to the amount of time any model
> takes. A simple method to reduce the wall-clock time is taken seriously.
> The NOAA developers have asked a couple of questions about the FILL / NOFILL
> option (here we reference this UNIDATA page):
> In our model history volumes, if we only open the files for write, then the
> NOFILL option seems to be (not only) an OK choice, but perhaps one that
> provides a timing reduction. Our tests indicate that a write with the NOFILL
> option is a 1x write, compared to a 2x write + 1x read when doing the FILL
> option. Would you please verify this for us? We think that we would like to
> set the NOFILL option as the default option in WRF, to benefit our entire
> community. Would you please comment on our possible plans in this regard to
> help us know if this is the right move. All of our data has TIME as the
> unlimited dimension. Other than the HDF5 compression capabilities from
> NETCDF4, we are a pretty vanilla user group of NetCDF.
I agree that setting the NOFILL option will save time in writing your model
and would be a reasonable default. The issue is whether the performance
any problems that might occur in determining what data was written and what
essentially garbage values in case you write the history data variables in a
used to define the variables in creating the file. For example, if you write
the last record
variable first and the writing is interrupted, then the values of the previous
in that record could be arbitrary. But if you log the creation of history
volumes so you know
when the writing is complete and buffers are flushed, either by a close or sync
call, that seems
like it shouldn't be a problem.
One other issue is a potentially serious bug related to use of nofill mode and
a specific pattern
of writing data in versions of netCDF before version 4.1.3 (released in June
[netcdfgroup] Important: potential file corruption using NOFILL mode
If you're sure that WRF users use version 4.1.3 or greater of the netCDF C
library, or that
WRF doesn't meet the conditions described above that trigger the bug, then I
think it's safe.
There was also a less serious bug with use of fill mode in netCDF versions
released in May 2012:
Fixed turning off fill values in HDF5 layers when NOFILL mode is set in
> Secondly, we are interested in knowing about the statement:
> The use of this feature may not be available (or even needed) in future
> releases. Programmers are cautioned against heavy reliance upon this feature.
> If we get timing reductions, we would like to rely on the continuing
> availability of this feature.
I think that statement was overly cautious, because we didn't know whether we
nofill mode in future versions. Now we're committed to backwards compatibility
features, and I don't anticipate that support for nofill mode would be dropped.
software layer used by netCDF-4 also supports fill values and
> Thanks in advance for your time and explanations.
You're welcome, I hope this helps.
> David Gill
> Software Engineer
> office: (303) 497 8162
> fax: (303) 497-8171
> Mailing address:
> 3450 Mitchell Lane
> Boulder, CO 80301
Russ Rew UCAR Unidata Program
Ticket ID: IJQ-160428
Department: Support netCDF