[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 950602: ncattput: NC_new_array: Not enough space

>Organization: BMRC
>Keywords: 199506020618.AA12340 netCDF ncapt

Hi Asri,

> Thank you for you suggestion. We ran our model in a higher memory allocation
> queue and the job was succesful.
>     We were using a 15 megaword queue on our Cray YMP and the next step 
> available was to move to a 28 megaword queue which made the elapse time for
> the job an order of magnitude longer. Do you know if changing the netCDF 
> file structure can reduce memory requirements?
>     Currently we have substantial amount of data defined as global 
> attributes. Will dividing them by making them attributes of several 
> different variable make any difference?

I did not appreciate from your original question that the problem with
running out of memory was occurring on a Cray.  This makes a difference,
because the netCDF library is written in C, the Fortran interface is
just a thin layer on top of the C interface that makes the calls follow
Fortran conventions, and the Cray Fortran interface differs from that for
other platforms.

When the Fortran interface for the Cray was developed, the Cray did not have
any Fortran data types such as BYTE, INTEGER*1, or INTEGER*2 corresponding
to the netCDF data types NCBYTE and NCSHORT.  Consequently, the Fortran
interface allocates extra space in memory to just copy arrays of bytes and
shorts from the C interface to Fortran integer arrays.  The way it does this
is rather crude, just calling malloc() to allocate all the space needed for
each I/O operation, rather than doing the packing and unpacking in
reasonably-sized chunks.

The Fortran interface also copies strings to privately-allocated space,
because it has to append a trailing "\0" to such strings for the C interface
to be able to deal with them.  Thus when you do an ncapt() call with a large
string, memory is allocated and a copy is made just to append the NULL
character at the end.

If this is the cause of the memory problems you are seeing, it would
certainly be possible to improve the strategies used in the Fortran
interface to make them smarter about using a local statically allocated
array and doing the conversion in chunks instead of all at once.  Two other
alternatives are to use the C interface to avoid the extra data copies, or
to see if the Cray Fortran compiler now supports something equivalent to the
BYTE, INTEGER*1, or INTEGER*2 declarations that are nonstandard but commonly
provided in other compilers.  If Cray Fortran now supports such facilities,
the memory allocation and copying could be completely eliminated for the
Fortran interface for the Cray with a few changes to the
netcdf/fortran/jackets.src file in the source distribution.

______________________________________________________________________________

Russ Rew                                           UCAR Unidata Program
address@hidden                              http://www.unidata.ucar.edu