Hi Jordi,
Please, you can find inline some comments and questions.
On 19/06/2023 16:51, Jordi Domingo Ballesta wrote:
Dear TDS team,
I would like to know if it is possible to (pre-)create the aggregation
cache and make thredds load it, in order to speed up the first time a
dataset is requested.
To give a bit of context, our situation is the following:
- We have a big archive of 265TB of data and 5 million files,
distributed in 1000 datasets (aprox).
Pretty challenge!!
This is just one catalog with 1000 datasets in that? If this is the
case, that is a big catalog to be parsed by server and client. If not,
what is the granularity of your catalogs? how many catalogs? and/or how
many datasets per catalog? how many files per dataset/aggregation?
- These datasets are in NetCDF format (mostly v4, some v3).
OK
- We run TDS version 5.4.
OK
- We configured thredds to provide access to them via "http" and
"odap" services, both directly (with "datasetScan") and as aggregated
datasets.
Only DAP (i.e. odap) service it's appropriate for aggregations. HTTP and
DAP it's possible for files.
- The configuration needs to be updated regularly (at least every day)
as new files come while others are deleted.
- We have serious performance issues regarding the access of
aggregated datasets, especially the first time they are accessed.
yes!! this can be a challenge
In order to improve that, we tried configuring the catalogs with the
explicit list of files for each dataset, including the "ncoords"
field, or even the "coordValue" field with the time value of each file
(they are joinExisting aggregations based on time dimension). That
improved substantially the performance of the first access, but the
duration is still not "acceptable" by the users.
This the best first option.
What means "not acceptable" .... or more exactly what is "acceptable" ...
This first approach should be enough, because you are using netCDF3/4,
but I'm curious about how many files are in each "dataset".
I tried to pre-create the cache files in thredds/cache/aggNew/
directory with the same content as when they are created by thredds,
but it seems that thredds is ignoring them when loading, and just
recreating its own version again. I also noticed that the cache
database in thredds/cache/catalog/ directory plays a role as well, but
I do not understand the relation between that and the aggregation
cache files.
This could be a good approach but it's tricky ... I have never managed
to make it work, as you just learnt.
Anyway, do you recommend any practice in order to improve the
performance of thredds for the first time a dataset is accessed? Maybe
throwing a 1-time request for the time variables to each dataset in
order to force thredds to create and load the cache?
Because you want to roll-up the dataset, I would follow a different
approach. Create independent NCML files with-in aggregations (with
datasetScand, for example, or explicit files), independent from the
catalog file, and pre-fill all metadata and coordinates. From catalog
files, you only have to refer those ncml files, or add a datasetScan
From my experience this the best approach because you will have better
"control" what happens with aggregations.
Please let me know if you need some clarifications.
Antonio
--
Antonio S. Cofiño
Instituto de Física de Cantabria (IFCA, CSIC-UC)
Consejo Superior de Investigaciones Científicas
http://antonio.cofino.es
#PublicMoneyPublicCode
#DocumentFreedomDay
Your help is very appreciated. Many thanks!
Kind regards,
*Jordi Domingo*
Senior software engineer
Lobelia Earth, S.L.
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web. Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.
thredds mailing list
thredds@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:https://www.unidata.ucar.edu/mailing_lists/
--
Antonio S. Cofiño
Instituto de Física de Cantabria (IFCA, CSIC-UC)
Consejo Superior de Investigaciones Científicas
http://antonio.cofino.es
#PublicMoneyPublicCode
#DocumentFreedomDay
