Re: THREDDS update

• To: Ken Keiser <keiser@xxxxxxxxxxxx>
• Subject: Re: THREDDS update
• From: Ethan Davis <edavis@xxxxxxxxxxxxxxxx>
• Date: Fri, 07 Feb 2003 13:23:41 -0700

Hi Ken, all,

I think you tried the generator back when it was using XSLT. The current
version is using the catalog library that John wrote which uses JDOM. I haven't
tested it on large collections like yours. But since JDOM creates an in-memory
document, I expect there will be similar memory issues. However, I suspect that
the JDOM version will be a bit less memory intensive than the XSLT version.
Though probably not enough to help with your large data collection.

I have two possible solutions to suggest for these situations. First, the
catalog could be split into smaller catalogs by using catalogRef elements. For
instance, you could have a top level catalog that showed the hierarchical
structure of your data collection but all the leaf nodes were catalogRefs
pointing to individual catalogs for each lower level collection. However, the
generator doesn't yet support yet the use of catalog refs. So, you would have to
run the generator for each catalogRef directory.

Second, we are working on Data Query Capability (DQC) documents for the case
were a single level of the hierarchy contains large numbers of datasets;
catalogRefs just won't help. Besides the memory issue with the generator, a
parser can get bogged down and, more importantly, a user will struggle to sort
through a huge list of datasets. We're seeing this with our radar data which has
several products coming in every second. Basically, the DQC allows us to
describe a set of allowed queries. For instance, with our radar dataset we can
let the user ask for the last hour of base reflectivity data from one specific
station. A catalog listing just those datasets that satisfy the request is
returned.

There's more on our DQC work on the THREDDS tech page:

http://www.unidata.ucar.edu/projects/THREDDS/tech/#DQC

Sorry to be so long winded. Hope this has been useful information.

Ethan

Ken Keiser wrote:
> 
> At the time it was memory errors, but as I said that was several months
> ago and I'm sure there are newer versions we have not tried since.
> 
> -----Original Message-----
> From: Robert Casey [mailto:rob@xxxxxxxxxxxxxxxxxxx]
> Sent: Tuesday, February 04, 2003 9:58 AM
> To: Ken Keiser
> Cc: Ben Domenico; thredds@xxxxxxxxxxxxxxxx
> Subject: RE: THREDDS update
> 
> > number of years.  The generator choked on the large number of
> > directories and granules.  We will download the latest version and try
> 
>         What kind of error did you get?  Array out of bounds?  Out of
> Memory
> error?
> 
> ------------------------------------------------------------------------
> Robert Casey, Software Engineer         web:   www.iris.washington.edu
> IRIS Data Management Center             email: rob@xxxxxxxxxxxxxxxxxxx
> 1408 NE 45th St.                        phone: (206) 547-0393 x117
> Seattle, WA 98105                       fax:   (206) 547-1093
> ------------------------------------------------------------------------

-- 
Ethan R. Davis                       Telephone: (303) 497-8155
Software Engineer                    Fax:       (303) 497-8690
UCAR Unidata Program Center          E-mail:    edavis@xxxxxxxx
P.O. Box 3000
Boulder, CO  80307-3000              http://www.unidata.ucar.edu/

• References:
  • RE: THREDDS update
    • From: Robert Casey [mailto:rob