That sounds like it, ncdump on 1 file shows "time = UNLIMITED ; // (8
currently)" it's kind of unexpected that these 8 values not be in a
contiguous array! Oh well. Thanks for clarifying. This is simulation
output, so our options may be limited. I will be sure to mention this to
the scientists. Hopefully they can write them as a fixed dimension.
On 06/02/2016 01:24 PM, Bowman, Kenneth P wrote:
Hi Burlen,
If time is your unlimited (record) dimension, then the time values are
scattered through the 433 MB file. That is true for any variables
that have a time dimension. To read the time variable, the netCDF
library has to jump through the file and collect the values.
The longitude variable is contiguous in memory and can be read quickly.
If you know the number of time steps in the file before you write the
file, you can change the unlimited time dimension to a fixed
dimension. Then something dimensioned by (only) time will be
contiguous in memory.
Or you can rewrite the files with fixed dimensions. That read
performance penalty is one of the tradeoffs of having the flexibility
of an unlimited dimension.
Good luck!
Ken
Date: Thu, 2 Jun 2016 12:41:53 -0700
From: Burlen Loring <bloring@xxxxxxx <mailto:bloring@xxxxxxx>>
To: Tom Fogal <tfogal@xxxxxxxxxxxx <mailto:tfogal@xxxxxxxxxxxx>>,
netcdfgroup@xxxxxxxxxxxxxxxx <mailto:netcdfgroup@xxxxxxxxxxxxxxxx>
Subject: Re: [netcdfgroup] abysmal performance
Message-ID: <961631fd-2aad-d348-ce1d-8a70a9e67287@xxxxxxx
<mailto:961631fd-2aad-d348-ce1d-8a70a9e67287@xxxxxxx>>
Content-Type: text/plain; charset=windows-1252; format=flowed
Hi Tom,
That's not an option, and it has it's own issues. for example if file
size exceeds the size of a tape drive we can't archive it. Beside it
doesn't seem like a lustre metadata issue, open is relatively fast, like
0.096 sec. and wouldn't explain why reading the time dimension with only
8 values takes on the order of 1 sec while reading the lon dimension
with 1152 values takes on the order of 1e-4 sec. ?
Burlen
-----------------------------------------------------------------------------
Dr. Kenneth P. Bowman 1014A Eller Building
David Bullock Harris Professor of Geosciences 979-862-4060
Department of Atmospheric Sciences 979-862-4466 fax
Texas A&M University
3150 TAMU
College Station, TX 77843-3150
_http://atmo.tamu.edu/people/faculty/bowmankenneth.html_
