[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #UNH-363541]: F90 netcdf 3.6.3 performance question

> I am at the Mesa Lab and would LOVE to talk to someone in person
> about this ...
> I am ready, willing and able to travel to FL.
> 
> I have a set of (non-netCDF) files to read. I grab a bit of each of
> these
> files and write to a set of netCDF files. If the netCDF file does not
> exist,
> I call an initialize routine which defines all the dimensions and
> variables,etc.,
> performs a 'snyc' and exits.  I have another routine that determines
> the current
> size of the unlimited dimension and does a series of 'put_var' calls,
> syncs, and exits.
> A third routine syncs and closes the netcdf file.
> 
> If the netCDF file exists and has an unlimited dimension length
> greater than 0 ...
> it takes about 3 seconds to fill what I need to fill - which is fine.
> It's the
> FIRST fill that kills me. Writing EXACTLY the same size 'stuff' takes
> about
> 10 minutes. I can NOT figure out why the first write takes so long.
> 
> This is on a new (less than 1 yr old) Mac compute-server with the
> Intel 11 Fortran
> compiler ...
> 
> pseudo code follows:
> 
> Nepochs = 12
> ObsFileLoop : do ifile=1, 10000
> EpochLoop : do iepoch = 1, Nepochs
> 
> InitializeNetCDF() ! there will be Nepochs different netCDF
> files ...
> ! if it exists already, check for
> compatibility ...
> 
> extract a subset of some data and
> WriteNetCDF()
> 
> CloseNetCDF()
> 
> enddo EpochLoop
> enddo ObsFileLoop
> 
> 
> The first time I run the code, it takes almost no time to
> IntializeNetCDF,
> a miserable long time (10 minutes?) to WriteNetCDF, and no time to
> CloseNetCDF.
> The second time I run the code with the same input files (leaving the
> newly-created netCDF files in place), it takes no time to
> InitalizeNetCDF,
> and about 3 seconds to WriteNetCDF. The netCDF files look as I expect
> them to look ...
> 
> I have attached copies of both routines and a dump of the netCDF
> header - in hopes someone will point out the error of my ways. Thanks
> -- Tim
> 
> P.S. nc_check() is a routine that ultimately calls a 'check'-like
> routine but also
> appends info about which variable/netcdf operation is choking. All of
> the die if they
> encounter an error condition.
> 

Howdy Tim!

As you note the data in the file are filled in with fill values when you create 
the variables. This can be turned off by changing the fill mode of the 
variables. Use the nf90_def_var_fill function, after your nf90_def_var 
function. It is documented here:
http://www.unidata.ucar.edu/software/netcdf/docs/netcdf-f90/NF90_005fDEF_005fVAR_005fFILL.html#NF90_005fDEF_005fVAR_005fFILL

So where you have a def_var, add a def_var_fill to turn off its fill mode.

If this does not address your performance problem, let me know.

Thanks,

Ed


Ticket Details
===================
Ticket ID: UNH-363541
Department: Support netCDF
Priority: Normal
Status: Closed