Hi Roland:
Before I respond. can you give a rough idea of your metric for "big" is (or
how many dataset elements and catalogRefs).
Thanks,
-Roy
On Jan 4, 2011, at 8:35 AM, Roland Schweitzer wrote:
> Thanks John. Among the groups we collaborate with there are some folks that
> are quite concerned about the scaling issue. Personally, my direct
> experience at this point that indicates that the performance is just fine (at
> least so far) even with our largest catalogs.
>
> What's the experience of the list? Are folks seeing unacceptable TDS
> initialization because of time spend reading catalogs? The thread from John
> Maurer about aggregation access issues notwithstanding.
>
> Roland
>
> On 01/03/2011 07:34 PM, John Caron wrote:
>> On 1/3/2011 10:53 AM, Roland Schweitzer wrote:
>>> Hi,
>>>
>>> We're starting to put together some "big" server-side configuration
>>> catalogs (both with "lots" of dataset elements and "lots" of catalogRef
>>> elements). We are wondering about the process TDS goes through to read the
>>> catalog when is starts. What gets cached? Does it have a way to know a
>>> referenced catalog is unchanged? When do referenced catalogs get scanned?
>>> And so on.
>>>
>>> Is there some documentation or a flow chart on how TDS initializes itself?
>>>
>>> Thanks,
>>> Roland
>>>
>>> _______________________________________________
>>> thredds mailing list
>>> thredds@xxxxxxxxxxxxxxxx
>>> For list information or to unsubscribe, visit:
>>> http://www.unidata.ucar.edu/mailing_lists/
>>
>> Hi Roland:
>>
>> The sad answer is theres not much documentation. Weve been on the verge of
>> redoing the initialization sequence for a few years now, so weve been
>> waiting so we can document the clean, cool refactor instead of the crufty,
>> lame current one.
>>
>> Anyway, the TDS reads in all the config catalogs at startup. It caches all
>> of them, and uses the "expires" attribute on the catalog to decide if/when
>> it needs to reread a catalog. It needs to read all catalogs, including
>> catalogRef, because it has to know what the possible dataset URLs are, and
>> there is no contract that a client has to read a catalog first.
>>
>> Obviously this doesnt scale forever. Ethan can probably fill in some details.
>>
>> see:
>> http://www.unidata.ucar.edu/projects/THREDDS/tech/catalog/v1.0.2/InvCatalogSpec.html#catalog
>>
>>
>> John
>>
>> _______________________________________________
>> thredds mailing list
>> thredds@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>
> _______________________________________________
> thredds mailing list
> thredds@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
**********************
"The contents of this message do not reflect any position of the U.S.
Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097
e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/
"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
