[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 20050427: Is there a way to read data "raw" and skip the swap calls

>To: address@hidden
>From: "Greg Sjaardema" <address@hidden>
>Subject: Is there a way to read data "raw" and skip the swap calls
>Organization: Sandia.gov
>Keywords: 200504272003.j3RK3Ev2005395

Hi Greg,

> We have a finite element results database which is based on netcdf. 
> Many of the analysis codes using this database run in parallel and are
> currently set up to write a single file per processor. There is then a
> second program (epu) which reads each of these individual files and
> merges them into a single file.  (We realize that this is inefficient in
> many cases and have other solutions in the works...).  The number of
> files in a set range from 2 up to ~1500 and the concatenated file sizes
> range up to 100s of gigabytes.
> 
> We have heavily optimized the epu program and are now to the point
> where, on Linux, the swapn8b or swapn4b  routines are responsible, in
> some cases, for more than 70% of the execution time.  The basic
> operation, once all of the mappings are set up is to read a variable
> from each processor file and map it into the correct location in the
> global array and then output the global array.  Since there is no time
> at which we actually examine the data during this process, we would like
> to find a way to eliminate the 70% penalty.
> 
> So, the question is whether there is a way to read a netcdf field "raw"
> and just get the bytes as they are stored on the disk.  We would then
> map this data into the global array and write it out also in a "raw"
> mode.  I've looked through the api documentation and don't see where
> this is possible, but thought I might have overlooked something.
> 
> If this isn't possible, we have thought of creating variants of the
> ncx_getn_float_float and ncx_putn_float_float (also double) routines
> which would skip the swap and just treat the data as a bunch of raw bytes...
> 
> We have had some of these jobs with estimated completion times of >40
> hours which has been dropped to ~10 hours based on some optimizations;
> but it would still be nice to drop from 10 hours down to 2-3 hours if
> possible...
> 
> Any suggestions are welcomed.

It's not possible through the API, as you've correctly determined.

Your idea of rewriting the ncx_getn_float_float and
ncx_putn_float_float and the analogous double functions is also the
best idea we've been able to come up with for the current netCDF
release.  Steve Emmerson is somewhat skeptical that you'll be able to
save a lot of time doing this: he thinks after this optimization
you'll find that instead of spending 70% of the time in swapn8b or
swapn4b you'll be spending a similar percentage of the time in
memcpy() (but that's just a guess).

We also have nonserious suggestions like using a big-endian machine,
but I'm sure that's not feasible at this point.

When we have a beta of netCDF-4 with parallel I/O available, it would
be interesting to know if it would save any time by writing only one
output file from all the processors concurrently and not having to
post-process all those files.  NetCDF-4 uses HDF5 and its parallel I/O
on top of MPI-IO, and we don't have much idea yet how this performs.
It still writes things out in XDR order, so it would byte swap on
little-endian platforms.

--Russ