Hi Roland: Before I respond. can you give a rough idea of your metric for "big" is (or how many dataset elements and catalogRefs). Thanks, -Roy On Jan 4, 2011, at 8:35 AM, Roland Schweitzer wrote: > Thanks John. Among the groups we collaborate with there are some folks that > are quite concerned about the scaling issue. Personally, my direct > experience at this point that indicates that the performance is just fine (at > least so far) even with our largest catalogs. > > What's the experience of the list? Are folks seeing unacceptable TDS > initialization because of time spend reading catalogs? The thread from John > Maurer about aggregation access issues notwithstanding. > > Roland > > On 01/03/2011 07:34 PM, John Caron wrote: >> On 1/3/2011 10:53 AM, Roland Schweitzer wrote: >>> Hi, >>> >>> We're starting to put together some "big" server-side configuration >>> catalogs (both with "lots" of dataset elements and "lots" of catalogRef >>> elements). We are wondering about the process TDS goes through to read the >>> catalog when is starts. What gets cached? Does it have a way to know a >>> referenced catalog is unchanged? When do referenced catalogs get scanned? >>> And so on. >>> >>> Is there some documentation or a flow chart on how TDS initializes itself? >>> >>> Thanks, >>> Roland >>> >>> _______________________________________________ >>> thredds mailing list >>> thredds@xxxxxxxxxxxxxxxx >>> For list information or to unsubscribe, visit: >>> http://www.unidata.ucar.edu/mailing_lists/ >> >> Hi Roland: >> >> The sad answer is theres not much documentation. Weve been on the verge of >> redoing the initialization sequence for a few years now, so weve been >> waiting so we can document the clean, cool refactor instead of the crufty, >> lame current one. >> >> Anyway, the TDS reads in all the config catalogs at startup. It caches all >> of them, and uses the "expires" attribute on the catalog to decide if/when >> it needs to reread a catalog. It needs to read all catalogs, including >> catalogRef, because it has to know what the possible dataset URLs are, and >> there is no contract that a client has to read a catalog first. >> >> Obviously this doesnt scale forever. Ethan can probably fill in some details. >> >> see: >> http://www.unidata.ucar.edu/projects/THREDDS/tech/catalog/v1.0.2/InvCatalogSpec.html#catalog >> >> >> John >> >> _______________________________________________ >> thredds mailing list >> thredds@xxxxxxxxxxxxxxxx >> For list information or to unsubscribe, visit: >> http://www.unidata.ucar.edu/mailing_lists/ > > _______________________________________________ > thredds mailing list > thredds@xxxxxxxxxxxxxxxx > For list information or to unsubscribe, visit: > http://www.unidata.ucar.edu/mailing_lists/ ********************** "The contents of this message do not reflect any position of the U.S. Government or NOAA." ********************** Roy Mendelssohn Supervisory Operations Research Analyst NOAA/NMFS Environmental Research Division Southwest Fisheries Science Center 1352 Lighthouse Avenue Pacific Grove, CA 93950-2097 e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address) voice: (831)-648-9029 fax: (831)-648-8440 www: http://www.pfeg.noaa.gov/ "Old age and treachery will overcome youth and skill." "From those who have been given much, much will be expected"