[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #CBQ-986702]: Why is nc_put_varm unbearable slow?



Florian-

The difference in speed is (probably)
due to the layout of the data on disk.
Netcdf-4 is built over the HDF5 library,
and HDF5 stores its data using a form of
b-tree. Netcdf classic, on the other hand
lays out the data in, more-or-less, a linear
form on the disk.

Hence, to write a single variable element in
netcdfclassic requires basically a disk seek
to the position of the element followed by a
write of the single data element. Accessing the data
thru a b-tree is inherently slowert. However, when writing
many consecutive values, HDF5 is substantially more efficient,
hence you should see much less difference when using nc_put_vara.

> ...Do you mean that the loop within nc_put_varm occurs when writing the 
> elements to
> disk? I would have expected that you allocate a dedicated buffer into which 
> the
> elements are presorted before being written to disk in chunks via 
> nc_put_vara. This 
> should be quite fast actually.
Potentially, but it is rather complex to do this. If you care to rewrite the
varm code to do it, we would certainly be grateful.



=Dennis Heimbigner
  Unidata


Ticket Details
===================
Ticket ID: CBQ-986702
Department: Support netCDF
Priority: Normal
Status: Closed


NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.