Re: [thredds] TDS initialization

Hi Roland:

Before I respond. can you give a rough idea of your metric for "big" is  (or 
how many dataset elements and catalogRefs).

Thanks,

-Roy

On Jan 4, 2011, at 8:35 AM, Roland Schweitzer wrote:

> Thanks John.  Among the groups we collaborate with there are some folks that 
> are quite concerned about the scaling issue.  Personally, my direct 
> experience at this point that indicates that the performance is just fine (at 
> least so far) even with our largest catalogs.
> 
> What's the experience of the list?  Are folks seeing unacceptable TDS 
> initialization because of time spend reading catalogs?  The thread from John 
> Maurer about aggregation access issues notwithstanding.
> 
> Roland
> 
> On 01/03/2011 07:34 PM, John Caron wrote:
>> On 1/3/2011 10:53 AM, Roland Schweitzer wrote:
>>> Hi,
>>> 
>>> We're starting to put together some "big" server-side configuration 
>>> catalogs (both with "lots" of dataset elements and "lots" of catalogRef 
>>> elements).  We are wondering about the process TDS goes through to read the 
>>> catalog when is starts.  What gets cached?  Does it have a way to know a 
>>> referenced catalog is unchanged?  When do referenced catalogs get scanned?  
>>> And so on.
>>> 
>>> Is there some documentation or a flow chart on how TDS initializes itself?
>>> 
>>> Thanks,
>>> Roland
>>> 
>>> _______________________________________________
>>> thredds mailing list
>>> thredds@xxxxxxxxxxxxxxxx
>>> For list information or to unsubscribe,  visit: 
>>> http://www.unidata.ucar.edu/mailing_lists/ 
>> 
>> Hi Roland:
>> 
>> The sad answer is theres not much documentation. Weve been on the verge of 
>> redoing the initialization sequence for a few years now, so weve been 
>> waiting so we can document the clean, cool refactor instead of the crufty, 
>> lame current one.
>> 
>> Anyway, the TDS reads in all the config catalogs at startup. It caches all 
>> of them, and uses the "expires" attribute on the catalog to decide if/when 
>> it needs to reread a catalog.  It needs to read all catalogs, including 
>> catalogRef, because it has to know what the possible dataset URLs are, and 
>> there is no contract that a client has to read a catalog first.
>> 
>> Obviously this doesnt scale forever. Ethan can probably fill in some details.
>> 
>> see:
>> http://www.unidata.ucar.edu/projects/THREDDS/tech/catalog/v1.0.2/InvCatalogSpec.html#catalog
>>  
>> 
>> John
>> 
>> _______________________________________________
>> thredds mailing list
>> thredds@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe,  visit: 
>> http://www.unidata.ucar.edu/mailing_lists/ 
> 
> _______________________________________________
> thredds mailing list
> thredds@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe,  visit: 
> http://www.unidata.ucar.edu/mailing_lists/ 

**********************
"The contents of this message do not reflect any position of the U.S. 
Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"