[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #IJQ-160428]: NetCDF no-fill option

Hi Dave,

> I am one of the group that provides support to the WRF model.  One of the 
> most used data formats for the WRF model is NetCDF, largely because of the 
> huge selection of post-processing tools that easily allows diagnostics and 
> visualization.  The WRF model is used in production at NOAA, and as such the 
> operations staff are always quite sensitive to the amount of time any model 
> takes.  A simple method to reduce the wall-clock time is taken seriously.
> 
> The NOAA developers have asked a couple of questions about the FILL / NOFILL 
> option (here we reference this UNIDATA page):
> http://www.unidata.ucar.edu/software/netcdf/docs/netcdf-c/nc_005fset_005ffill.html
> 
> In our model history volumes, if we only open the files for write, then the 
> NOFILL option seems to be (not only) an OK choice, but perhaps one that 
> provides a timing reduction.  Our tests indicate that a write with the NOFILL 
> option is a 1x write, compared to a 2x write + 1x read when doing the FILL 
> option.  Would you please verify this for us?  We think that we would like to 
> set the NOFILL option as the default option in WRF, to benefit our entire 
> community.  Would you please comment on our possible plans in this regard to 
> help us know if this is the right move.  All of our data has TIME as the 
> unlimited dimension.  Other than the HDF5 compression capabilities from 
> NETCDF4, we are a pretty vanilla user group of NetCDF.

I agree that setting the NOFILL option will save time in writing your model 
history volumes,
and would be a reasonable default.  The issue is whether the performance 
benefit outweighs
any problems that might occur in determining what data was written and what 
data are
essentially garbage values in case you write the history data variables in a 
different order 
used to define the variables in creating the file.  For example, if you write 
the last record
variable first and the writing is interrupted, then the values of the previous 
record variables
in that record could be arbitrary.  But if you log the creation of history 
volumes so you know 
when the writing is complete and buffers are flushed, either by a close or sync 
call, that seems
like it shouldn't be a problem.

One other issue is a potentially serious bug related to use of nofill mode and 
a specific pattern
of writing data in versions of netCDF before version 4.1.3 (released in June 
2011):

  [netcdfgroup] Important: potential file corruption using NOFILL mode
  
http://www.unidata.ucar.edu/mailing_lists/archives/netcdfgroup/2011/msg00137.html

If you're sure that WRF users use version 4.1.3 or greater of the netCDF C 
library, or that
WRF doesn't meet the conditions described above that trigger the bug, then I 
think it's safe.

There was also a less serious bug with use of fill mode in netCDF versions 
before 4.2,
released in May 2012:

  Fixed turning off fill values in HDF5 layers when NOFILL mode is set in 
netCDF-4 API
  https://bugtracking.unidata.ucar.edu/browse/NCF-151

> Secondly, we are interested in knowing about the statement:
> The use of this feature may not be available (or even needed) in future 
> releases. Programmers are cautioned against heavy reliance upon this feature.
> If we get timing reductions, we would like to rely on the continuing 
> availability of this feature.

I think that statement was overly cautious, because we didn't know whether we 
could support
nofill mode in future versions.  Now we're committed to backwards compatibility 
for such
features, and I don't anticipate that support for nofill mode would be dropped. 
 The HDF5
software layer used by netCDF-4 also supports fill values and 

> Thanks in advance for your time and explanations.

You're welcome, I hope this helps.

--Russ

> Dave
> 
> 
> David Gill
> Software Engineer
> address@hidden
> office: (303) 497 8162
> fax: (303) 497-8171
> Mailing address:
> 3450 Mitchell Lane
> Boulder, CO 80301
> 
> 
> 
> 
Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: IJQ-160428
Department: Support netCDF
Priority: Normal
Status: Closed