[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Fwd: netcdf4 vs. netcdf3

Hi Arlindo,

Ed forwarded your question to me.

>   Back in September I sent you the e-mail below. It turns out that
> during the HDF conference our ops guys got the word that version of
> HDF-5 you are basing your NC-4 implementation is still "alpha", in
> the sense that  they may still change the internal format between
> now and the final release. Therefore, it does not appear to be
> sensible to base a full reanalysis data set on a format that may
> not  be quite standard when the dataset is out - rats! So, we are
> back in the HDF-4/NetCDF-3 world.
> 
>   To refresh your mind, it costs a lot more to write bigendian data
> on the Altix. Be it in FORTRAN binary (using the "-convert
> bigendian" option of the compiler) or in NetCDF-3 or NetCDF-4
> (setting the bigendian option), it is almost one order of magnitude
> more expensive to write bigendian than littleendian. The problem has
> been reported to Intel but they have not been very responsive.
> 
>  On the fortran end, doing the byte swap ourselves kind of solves
> the problem. Here is the question for you: In NetCDF-3/4, where is
> the littleendian -> bigendian convertion taking place? Are you
> relying on any system provided rpc/xdr library to do the conversion
> or simply doing a byteswap plus a fwrite? If I could patch NetCDF-3
> to avoid a buggy system library (while writing standard nc-3
> bigendian files) it could save us lots of unncessary postprocessing.
> 
>   Any help greatly appreciated.
>     Thanks,
>        Arlindo
> 
> ---------- Forwarded message ----------
> From: Arlindo da Silva <address@hidden>
> Date: Sep 14, 2006 10:34 AM
> Subject: Fwd: netcdf4 vs. netcdf3
> To: Ed Hartnett <address@hidden>
> Hi Ed,
>     We were able to run a test comparing NetCDF-3/4 on the Altix. The
> output file consisted of a number of 2D/3D global fields at 0.25
> degree resolution. The file size was about 350 MB.  Using hdf-5 as the
> output format in NetCDF-4 we got a write-out speed up of  28.9s/3.9s =
> 7.4!!!
>    We haven't tried this on other platforms, but I suspect this is
> Altix specific, and is related to big-endian conversion issue I
> mentioned to you before. This is reason enough for us to switch to
> NetCDF-4 in our next re-analysis starting later this year.

It may not be Altix specific.  When we first developed netCDF, our
tests showed that the amount of time spent byte-swapping was swamped
by other operations involved in disk I/O, so it was not an issue.
However, lately we've heard from several users on little-endian
architectures that byte-swapping is where most of the time is spent
for some netCDF-3 programs, especially those that just copy or merge
netCDF data, swapping it on reads and then again on writes.

Where this all happens in netCDF-3 is in the source file libsrc/ncx.c
(which is actually generated by the m4 macroprocessor from
libsrc/ncx.m4, where we recommend making edits so they won't get
overwritten).

The code between

  #ifndef WORDS_BIGENDIAN

and

  #endif /* LITTLE_ENDIAN */

is where all the swapping takes place, in functions swapn2b, swapn4b,
and swapn8b (for 2-byte, 4-byte, and 8-byte numerical types,
respectively).  In the latest beta release that Ed will be making
available very soon, there is some optimization of this code that
saves time by unrolling loops.

Unfortunately this is not calling a "buggy system library" to do the
byte swap, so it's probably not of much help.  If there were a standard
system library function for byte-swapping an array of values, I think
we should be using it instead, because it might be able to exploit
fast hardware, but I don't know of such a library function.  If you
know any other general way to improve the speed of this code, we'd be
interested.

By defining these functions to do nothing you would save all the
swapping time.  The resulting library would no longer write netCDF
files that could be read by others, but would instead write (and read)
a "little-endian" variant of netCDF.  If you're only using these files
for internal purposes (e.g. restart files for a long-running model),
that might be just what you need, but of course it wouldn't be good
for archiving, if you ever want anyone else to access the data.

--Russ