[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
Re: netcdf write performance (fwd)
- Subject: Re: netcdf write performance (fwd)
- Date: Mon, 24 Nov 2003 09:30:36 -0700
------- Forwarded Message
Date: Mon, 24 Nov 2003 08:46:04 -0700
From: Tim Hoar <address@hidden>
To: Russ Rew <address@hidden>
Subject: Re: netcdf write performance
Russ --
we'd like to keep the "unknown" number of timesteps, so I will
have to pay attention to allocating a fixed size time array and
"growing" it if,when we reach the end.
Not terribly difficult, just not as tidy as using the netcdf time
coordinate variable, and introduces the possibility that the time
coordinate and the internal time array can diverge ...
Thanks --
Tim
> Date: Fri, 21 Nov 2003 12:59:35 -0700
> From: Russ Rew <address@hidden>
> To: Tim Hoar <address@hidden>
> Cc: Jeffrey Anderson <address@hidden>, address@hidden
> Subject: Re: netcdf write performance
>
> >To: address@hidden
> >From: Tim Hoar <address@hidden>
> >Subject: Re: 20031120: netcdf write performance
> >Organization: UCAR
> >Keywords: 200311202353.hAKNrmEH017770
>
> Hi Tim,
>
> > [for Russ' benefit: I have a (random) time and want to either add to the
> > existing unlimited dimension or insert at the appropriate (existing) time.
> > Using nc_inq_dimlen is insufficient, since I need to know the corresponding
> > time]
> >
> > If I read the netcdf coordinate variable for time (the unlimited dimension)
> > each time I encounter the above situation, things grow as follows:
> >
> > Running case 2000 at Thu Nov 20 16:16:32 MST 2003
> > file_name = Mydata
> > dim1 = 10
> > dim2 = 100
> > nT = 2000
> > t = 0 time is 1380838617
> > Model attributes written, netCDF file synched ...
> > t = 100 elapsed time is 1303
> > t = 200 elapsed time is 1488
> > t = 300 elapsed time is 1835
> > t = 400 elapsed time is 2182
> > t = 500 elapsed time is 2535
> > t = 600 elapsed time is 2889
> > t = 700 elapsed time is 3264
> > t = 800 elapsed time is 3618
> > t = 900 elapsed time is 3990
> > t = 1000 elapsed time is 4690
> > t = 1100 elapsed time is 4834
> > t = 1200 elapsed time is 5194
> > t = 1300 elapsed time is 5594
> > t = 1400 elapsed time is 5997
> > t = 1500 elapsed time is 6395
> > t = 1600 elapsed time is 6798
> > t = 1700 elapsed time is 7201
> > t = 1800 elapsed time is 7900
> > t = 1900 elapsed time is 8004
> > t = 2000 elapsed time is 8399
> > 2.548u 6.880s 0:09.42 100.0% 0+0k 0+0io 253pf+0w
> >
> >
> > The elapsed time is the result of the Fortran90 intrinsic "system_clock"
> > and has terrific units -- some unknown integer representation of the system
> > clock. What I have printed is the time spent since the last print (i.e. if
> > the system is linear, the elapsed time printed would be constant).
> >
> > However,
> >
> > If I use exactly the same methodology but keep a local array of the times
> > present in the netcdf file (i.e. I replace the call to read the current
> > netcdf time variable with a local time array) -- the code performs linearly:
> >
> > Running case 2000 at Thu Nov 20 16:12:35 MST 2003
> > file_name = Mydata
> > dim1 = 10
> > dim2 = 100
> > nT = 2000
> > t = 0 time is 1378473155
> > Model attributes written, netCDF file synched ...
> > t = 100 elapsed time is 969
> > t = 200 elapsed time is 968
> > t = 300 elapsed time is 967
> > t = 400 elapsed time is 967
> > t = 500 elapsed time is 966
> > t = 600 elapsed time is 970
> > t = 700 elapsed time is 968
> > t = 800 elapsed time is 968
> > t = 900 elapsed time is 967
> > t = 1000 elapsed time is 968
> > t = 1100 elapsed time is 968
> > t = 1200 elapsed time is 971
> > t = 1300 elapsed time is 968
> > t = 1400 elapsed time is 969
> > t = 1500 elapsed time is 972
> > t = 1600 elapsed time is 971
> > t = 1700 elapsed time is 968
> > t = 1800 elapsed time is 968
> > t = 1900 elapsed time is 968
> > t = 2000 elapsed time is 968
> > 0.380u 1.562s 0:01.94 100.0% 0+0k 0+0io 240pf+0w
> >
> > This includes the fact that I write to the netcdf time array in
> > exactly the same manner as the non-linear response case.
> >
> > There is no error in the slow case, just a design decision that
> > seems perfectly logical yet results in very slow code. In order to
> > achieve linear performance in the production code, I will have to
> > jump through some hoops.
>
> With your current structure, reading the entire time coordinate
> variable involves reading a single value from each record, so if you
> have 2000 times, that's 2000 disk reads to get a table of 2000 values.
> All the slowdown is due to these reads of an ever-increasing number of
> records. As you have discovered, if you cache the times in a table in
> memory and replace the reads of the time variable with accesses to
> this table, there is no slowdown.
>
> This may be obvious and unappealing, but as another workaround you
> could store an auxiliary fixed size variable containing a copy of the
> time variable with enough capacity to hold all the times you will ever
> add to one dataset. If you update this whenever you add a new time
> and read from it rather than from the time record variable, you can
> get all the time values in only a few disk accesses and things won't
> slow down significantly as you search through more times.
>
> --Russ
>
> _____________________________________________________________________
>
> Russ Rew UCAR Unidata Program
> address@hidden http://www.unidata.ucar.edu/staff/russ
>
## Tim Hoar, Associate Scientist email: address@hidden ##
## Geophysical Statistics Project phone: 303-497-1708 ##
## National Center for Atmospheric Research FAX : 303-497-1333 ##
## Boulder, CO 80307 http://www.cgd.ucar.edu/~thoar ##
------- End of Forwarded Message
