[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 19990406: Time to perform nc_close, enddef...

Chris,

> To: <address@hidden>
> From: address@hidden
> Subject: Time to perform nc_close, enddef...
> Organization: .
> Keywords: 199904061901.NAA12265

In the above message, you wrote:

>      I've noticed that the nc_close and nc_enddef functions seem to take a 
>      considerable amount of time to perform.  For example, we perform a 
>      variety of utilities on .CDF libraries, like ordering, sorting and 
>      re-structuring, as well as splitting and merging various libraries.  
>      If you watch the file sizes as these things execute, and throw in some 
>      appropriate flags to profile the execution, it is clear that probably 
>      90% or so of the overall execution time is used by these two 
>      functions.

I'm not surprised.  Each nc_redef/nc_enddef (or nc_close) pair performs
a copy of all the data in the file.  If you look at Section 9 (NetCDF
File Structure and Perormance) in the "NetCDF User's Guide for C" you'll
see why.

>      Is there a problem with our NetCDF implementation or is this normal?  
>      And if it is normal, is there any way around it?  We're trying to use 
>      these utilities in near real time, and these two functions seem to 
>      pose a big bottleneck. 

The workaround is to do as few nc_redef-s as possible -- preferably
none.  If you know what the final form of the dataset will be, then you
can define it once at the beginning of the process and have the
subprocesses fill-in the missing data as appropriate.

>      By the way, we're using version 3.3.1 (C interface) on Red Hat Linux 
>      5.1.  I can also provide you with more specific data and/or bits of 
>      source code, if necessary, but I'm guessing you may be familiar with 
>      this symptom already.
>      
>      Thanks very much for your help.
>      
>      Regards,
>      
>      Chris Raymond

--------
Steve Emmerson   <http://www.unidata.ucar.edu>