Dear TDS team,
I would like to know if it is possible to (pre-)create the aggregation
cache and make thredds load it, in order to speed up the first time a
dataset is requested.
To give a bit of context, our situation is the following:
- We have a big archive of 265TB of data and 5 million files, distributed
in 1000 datasets (aprox).
- These datasets are in NetCDF format (mostly v4, some v3).
- We run TDS version 5.4.
- We configured thredds to provide access to them via "http" and "odap"
services, both directly (with "datasetScan") and as aggregated datasets.
- The configuration needs to be updated regularly (at least every day) as
new files come while others are deleted.
- We have serious performance issues regarding the access of aggregated
datasets, especially the first time they are accessed.
In order to improve that, we tried configuring the catalogs with the
explicit list of files for each dataset, including the "ncoords" field, or
even the "coordValue" field with the time value of each file (they are
joinExisting aggregations based on time dimension). That improved
substantially the performance of the first access, but the duration is
still not "acceptable" by the users.
I tried to pre-create the cache files in thredds/cache/aggNew/ directory
with the same content as when they are created by thredds, but it seems
that thredds is ignoring them when loading, and just recreating its own
version again. I also noticed that the cache database in
thredds/cache/catalog/ directory plays a role as well, but I do not
understand the relation between that and the aggregation cache files.
Anyway, do you recommend any practice in order to improve the performance
of thredds for the first time a dataset is accessed? Maybe throwing a
1-time request for the time variables to each dataset in order to force
thredds to create and load the cache?
Your help is very appreciated. Many thanks!
Kind regards,
*Jordi Domingo*
Senior software engineer
Lobelia Earth, S.L.
