Hi Jeff:
BTW, the default buffer size in Structure.Iterator is 500K, which gives better
performance (I think) when the structure is actually a "psuedo-structure", that
is, stored by column rather than row.
I could try modifying the default buffer size of RAF, that might improve
performance on the server.
The trick is to have a wide range of queries to optimize over, and not just get
one case right at the expense of the whole.
Do you want to put together an interesting "load" on the server, maybe based on
a bunch of IDV tests or something?
Jeff McWhirter wrote:
What experience do you all have with the performance characteristics of
data access
with varying bufferesizes. I have found with the point obs that it
varies dramatically for remote
data sets. e.g, getting a data iterator with BUFFERSIZE using:
PointObsDataset.getDataIterator(BUFFERSIZE)
and reading this data set (~90000 obs,32 values)
http://lead4.unidata.ucar.edu:8080/thredds/dodsC/station/metar/20060716_metar.nc
I get:
buffer size:4096 Total time:41882
buffer size:8192 Total time:30348
buffer size:16384 Total time:25124
buffer size:32768 Total time:23919
buffer size:65536 Total time:26789
buffer size:131072 Total time:27540
It seems as though 32768 is a sweet spot. Any ideas why the higher
buffer sizes give worse performance?
Are you allocating the buffers repeatedly? (thus triggering worse GC
behavior?)
On the other hand reading a local file:
/upc/share/testdata/station/madis/20060615_1200
which has 23000 obs and 180 vars I get the exact opposite:
buffer size:4096 Total time:8557
buffer size:8192 Total time:8534
buffer size:16384 Total time:8307
buffer size:32768 Total time:11004
buffer size:65536 Total time:9247
buffer size:131072 Total time:9010
buffer size:262144 Total time:9032
-Jeff
===============================================================================
To unsubscribe thredds-dev, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html
===============================================================================
