Re: buffer size and performance

• Subject: Re: buffer size and performance
• From: John Caron <caron@xxxxxxxxxxxxxxxx>
• Date: Fri, 28 Jul 2006 08:42:51 -0600

Hi Jeff:

BTW, the default buffer size in Structure.Iterator is 500K, which gives better 
performance (I think) when the structure is actually a "psuedo-structure", that 
is, stored by column rather than row.

I could try modifying the default buffer size of RAF, that might improve 
performance on the server.

The trick is to have a wide range of queries to optimize over, and not just get 
one case right at the expense of the whole.

Do you want to put together an interesting "load" on the server, maybe based on 
a bunch of IDV tests or something?



Jeff McWhirter wrote:

What experience do you all have with the performance characteristics of data access with varying bufferesizes. I have found with the point obs that it varies dramatically for remote

data sets. e.g, getting a data iterator with BUFFERSIZE using:

PointObsDataset.getDataIterator(BUFFERSIZE)

and reading this data set (~90000 obs,32 values) http://lead4.unidata.ucar.edu:8080/thredds/dodsC/station/metar/20060716_metar.nc

I get:
buffer size:4096 Total time:41882
buffer size:8192 Total time:30348
buffer size:16384 Total time:25124
buffer size:32768 Total time:23919
buffer size:65536 Total time:26789
buffer size:131072 Total time:27540

It seems as though 32768 is a sweet spot. Any ideas why the higher buffer sizes give worse performance? Are you allocating the buffers repeatedly? (thus triggering worse GC behavior?)

On the other hand reading a local file:
/upc/share/testdata/station/madis/20060615_1200
which has 23000 obs and 180 vars I get the exact opposite:
buffer size:4096 Total time:8557
buffer size:8192 Total time:8534
buffer size:16384 Total time:8307
buffer size:32768 Total time:11004
buffer size:65536 Total time:9247
buffer size:131072 Total time:9010
buffer size:262144 Total time:9032



-Jeff

===============================================================================

To unsubscribe thredds-dev, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html

===============================================================================

• Follow-Ups:
  • Re: buffer size and performance
    • From: Don Murray