[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

RE: THREDDS update



Title: Re: THREDDS update
Ethan,
 
Thanks for the response.  I'm hoping to give the catalog generation another try, so I will look into your suggestions.  Our fallback is just to write some scripts to generate the xml for our catalog manually...doesn't look too bad, except that it wouldn't stay current with any changes you make for the schema I suppose.  We'll figure something out.  Still looking at catalog definitions for OGC services as well.
 
thanks,
 
Ken
-----Original Message----- Et
From: Ethan Davis [mailto:address@hidden]
Sent: Fri 02/07/2003 02:23 PM
To: Ken Keiser
Cc: Robert Casey; address@hidden
Subject: Re: THREDDS update

Hi Ken, all,

I think you tried the generator back when it was using XSLT. The current
version is using the catalog library that John wrote which uses JDOM. I haven't
tested it on large collections like yours. But since JDOM creates an in-memory
document, I expect there will be similar memory issues. However, I suspect that
the JDOM version will be a bit less memory intensive than the XSLT version.
Though probably not enough to help with your large data collection.

I have two possible solutions to suggest for these situations. First, the
catalog could be split into smaller catalogs by using catalogRef elements. For
instance, you could have a top level catalog that showed the hierarchical
structure of your data collection but all the leaf nodes were catalogRefs
pointing to individual catalogs for each lower level collection. However, the
generator doesn't yet support yet the use of catalog refs. So, you would have to
run the generator for each catalogRef directory.

Second, we are working on Data Query Capability (DQC) documents for the case
were a single level of the hierarchy contains large numbers of datasets;
catalogRefs just won't help. Besides the memory issue with the generator, a
parser can get bogged down and, more importantly, a user will struggle to sort
through a huge list of datasets. We're seeing this with our radar data which has
several products coming in every second. Basically, the DQC allows us to
describe a set of allowed queries. For instance, with our radar dataset we can
let the user ask for the last hour of base reflectivity data from one specific
station. A catalog listing just those datasets that satisfy the request is
returned.

There's more on our DQC work on the THREDDS tech page:

http://www.unidata.ucar.edu/projects/THREDDS/tech/#DQC

Sorry to be so long winded. Hope this has been useful information.

Ethan

Ken Keiser wrote:
>
> At the time it was memory errors, but as I said that was several months
> ago and I'm sure there are newer versions we have not tried since.
>
> -----Original Message-----
> From: Robert Casey [mailto:address@hidden]
> Sent: Tuesday, February 04, 2003 9:58 AM
> To: Ken Keiser
> Cc: Ben Domenico; address@hidden
> Subject: RE: THREDDS update
>
> > number of years.  The generator choked on the large number of
> > directories and granules.  We will download the latest version and try
>
>         What kind of error did you get?  Array out of bounds?  Out of
> Memory
> error?
>
> ------------------------------------------------------------------------
> Robert Casey, Software Engineer         web:   www.iris.washington.edu
> IRIS Data Management Center             email: address@hidden
> 1408 NE 45th St.                        phone: (206) 547-0393 x117
> Seattle, WA 98105                       fax:   (206) 547-1093
> ------------------------------------------------------------------------

--
Ethan R. Davis                       Telephone: (303) 497-8155
Software Engineer                    Fax:       (303) 497-8690
UCAR Unidata Program Center          E-mail:    address@hidden
P.O. Box 3000
Boulder, CO  80307-3000              http://www.unidata.ucar.edu/