THREDDS slow with lots of files aggregated?
- To: THREDDS <address@hidden>
- Subject: THREDDS slow with lots of files aggregated?
- From: "Bob Simons" <address@hidden>
- Date: Fri, 14 Jul 2006 11:52:43 -0700
We have a THREDDS server running on Linux here at ERD serving a lot of
data sets (http://oceanwatch.pfeg.noaa.gov:8081/thredds/catalog.html).
The data sets are aggregates, created by putting lots of individual
files in each data set's directory.
The problem is: I have noticed that the time to get data from 1 data
file in a dataset is roughly proportional to the number of files in the
directory. And access to data in directories with lots of files is very
Here are the results from a test in the order that the subtests were
done (in one run of the test program). The GAssta hday subtest was
added to test the theory that the number of files in the directory was
correlated with the thredds opendap response time.
AG ssta 3day: 190 files, 719 ms
CM usfc hday: 2138 files, 13359 ms
GA ssta hday: 1018 files, 7063 ms
MB chla 1day: 185 files, 625 ms
QN curl 8day: 537 files, 3141 ms
That looks like a great correlation to me.
Any idea why THREDDS is so slow at opening one file? Linux is slow with
so many files, but not close to this slow. Is there anything we can do
to Linux or THREDDS to improve this?
Satellite Data Product Manager
Environmental Research Division
NOAA Southwest Fisheries Science Center
1352 Lighthouse Ave
Pacific Grove, CA 93950-2079
<>< <>< <>< <>< <>< <>< <>< <>< <><
To unsubscribe thredds, visit:
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publicly available
through the web. If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.