[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: latest Catalog XML



Daniel Holloway wrote:
> 
> Ethan Davis wrote:
> 
> > Daniel Holloway wrote:
> > >
> > > ....
> > >
> > >     I've used the DTD listed above to create two examples of what
> > > I believe I could return in an XML-based dods-dir response.  The
> > > catalogs represent simple directory representations of dods-accessible
> > > data; level1.xml is a higher-level directory containing only subdirs
> > > and level2.xml is one of the subdirs containing the dods-accessible
> > > datafiles.
> > >
> > >     Both of these files validate, I get some warnings but mostly
> > > due to style issues, there are no errors.   These examples represent
> > > the minimum information that I could provide, there are ways I
> > > could augment this service to provide some of the additional fields
> > > represented in the DTD but this example is meant to illustrate a
> > > minimum set.
> > >
> > >     Points to note:
> > >
> > > 1: I'd need to provide 2 service elements, one for
> > > type = DODS, and one for type = Other, where Other represents
> > > the dods-dir service.   In many cases there will be both datafiles
> > > and subdirectories in a file system.   Do I need to have two
> > > service elements?   If so, in the 'Level2.xml' example how do
> > > I indicate which service to use without adding an explicit
> > > access element for each dataset element?
> >
> > In each dataset element you can indicate the service it uses with the
> > serviceName attribute; its value would be the same as the value of the
> > service
> > elements name attribute.
> >
> 
> OK
> 
> >
> > > 2:  I use 'collection' as a simple filesystem collection, there
> > > may not be any meaningful relation between the files contained
> > > in the directory other than they're accessible via the DODS DAP.
> > >
> > > 3:  Am I using 'catalogRef' correctly?  The intention is to
> > > indicate another collection level that should only be traversed
> > > at the user's request.
> >
> > catalogRef should reference a THREDDS catalog. Yours is referencing a
> > dods-dir
> > page. I've reworked and attached your examples (with fewer years in
> > level1.xml
> > and fewer example datasets in level2.1985.xml).
> 
> Actually I'm trying to reference a THREDDS catalog, that is, my intention
> is that the response from a dods-dir service request would be a valid THREDDS
> catalog and it may look as indicated in the original example.   Currently,
> the dods-dir service returns an html-encoded page, that service would be
> augmented to return a THREDDS catalog.  Granted a THREDDS catalog
> is meant to offer more than a simple filesystem view of the accessible files
> but the catalog DTD supports seems to support this use.

Sorry, my mistake. I didn't realize you meant for the dods-dir to return a
THREDDS catalog. That sounds good. Your original example should work fine. The
only change needed would be making the datasets reference a service.

> I may be misreading the examples which have been provided in earlier
> messages, but it seems that all of the 'hrefs' in the catalogRef examples
> use explict filenames, but as long as the response to the 'href' returns
> a valid THREDDS catalog representation, why can't the 'href' reference
> a service which returns such a catalog.   The use of 'service' in this
> case is somewhat different than the 'access' service that is encoded
> in the service element, though arguably there is the 'other' and 'catalog'
> types.
>
> If you allow this, then the catalog becomes much simpler for the dods-dir
> case, basically following my initial example.   Actually, I'm not sure how
> the THREDDS API or any other parsing application would know the
> difference in the href so long as it returned a valid xml representation.
> 

Yup. The 'href' can be any URL that returns a THREDDS catalog. In your top level
catalog you could even get rid of all the service elements because nothing in
that catalog references them.

> I think the example you provided below using both dataset and
> catalogRef elements to depict the filesystem layout is somewhat
> ambiguous since the relation between these two elements is
> implicit in this use, and not explicit in the DTD, such that any
> API and or parsing application would need to recognize this
> particular relationship.

Good point. Not a very clean representation.

> > 1) I changed the catalogRefs in level1.xml to point to the level2.*.xml
> > files.
> 
> A minor point but I'd prefer not to point to individual 'level#.xml' files in
> the references and just point to the service instance for invoking dods-dir
> on the underlying subdirs.  Not sure how to do that now.
> 
> >
> 
> >
> > 2) I added datasets in level1.xml that give access to the dods-dir pages.
> > 3) I added a serviceName attribute in the dataset elements in the
> > level2.*.xml
> > files.
> >
> > A few alternatives:
> >
> > 1) You could have just one level. Each year could be a collection that
> > contains
> > the individual datasets.
> 
> The catalog I'm representing with the dods-dir response is a view
> of the filesystem tree for an arbitrary branch or leaf of the
> filesystem, and for only dods-accessible datafiles at that branch
> or leaf.   I definitely don't want to populate the complete filesystem
> catalog at a high level but want the client app or UI to traverse
> the references to finally get the complete list of datafiles.  Our site
> has approx. 100K datafiles accessible via DODS, so I need to
> limit the amount of information I transmit to a reasonable amount
> unless explictly requested by the client.
> 
> >
> >
> > 2) Again, just one level. Each year is a dataset with the dods-dir access
> > and
> > contains sub-datasets for each individual dataset.
> >
> > I'm working on the THREDDS catalog generator tool. Currently I'm working on
> > expanding DODS file servers into THREDDS catalogs. After I get that working
> > I'd
> > like to work on crawling dods-dir pages. So, I'll be wanting to pick your
> > brains
> > on this stuff.
> >
> 
> My plan is to augment dods-dir to return an xml-encoded response, hopefully
> a THREDDS catalog representation, then you won't have to crawl dods-dir pages
> just traverse those catalog representations.   We could then discuss how to
> extend the dods-dir service, possibly to read some config.xml or other
> ancillary
> info sources on the server to provide the additional information needed to
> extend these simple filesystem catalog views into more functional catalog
> representations.

Sounds great. Sorry for the confusion. Your one step (at least) ahead of me.

Ethan


> >   ------------------------------------------------------------------------
> > <?xml version="1.0" encoding="UTF-8"?>
> > <!DOCTYPE catalog SYSTEM
> > "http://www.unidata.ucar.edu/projects/THREDDS/xml/InvCatalog.0.6d.dtd";>
> > <catalog name="htn_sst_decloud" version="0.6d">
> >    <collection name="htn_sst_decloud/">
> >       <service name="URI-DODS" serviceType="DODS"
> >                base="http://dods.gso.uri.edu/dods-3.2/nph-dods/"/>
> >       <service name="DODSdirectory" serviceType="Other"
> >                      base="http://dods.gso.uri.edu/dods-3.2/nph-dods/"/>
> >
> >       <dataset name="DODS_dir of htn_sst_decloud/1985/"
> >                serviceName="DODSdirectory"/>
> >       <catalogRef xlink:title="htn_sst_decloud/1985/"
> >                   xlink:href="./level2.1985.xml"/>
> >
> 
> >
> >       <dataset name="DODS_dir of htn_sst_decloud/1986/"
> >                serviceName="DODSdirectory"/>
> >       <catalogRef xlink:title="htn_sst_decloud/1986/"
> >                   xlink:href="./level2.1986.xml"/>
> >
> >    </collection>
> > </catalog>
> >
> >   ------------------------------------------------------------------------
> > <?xml version="1.0" encoding="UTF-8"?>
> > <!DOCTYPE catalog SYSTEM
> > "http://www.unidata.ucar.edu/projects/THREDDS/xml/InvCatalog.0.6d.dtd";>
> > <catalog name="htn_sst_decloud/1985" version="0.6d">
> >   <collection name="htn_sst_decloud/1985">
> >      <service name="URI-DODS" serviceType="DODS"
> >               base="http://dods.gso.uri.edu/dods-3.2/nph-dods/"/>
> >      <service name="DODSDirectory" serviceType="Other"
> >               base="http://dods.gso.uri.edu/dods-3.2/nph-dods/"/>
> >      <dataset name="k85001182828.htn_d" serviceName="URI-DODS"
> >               urlPath="htn_sst_decloud/1985/k85001182828.htn_d.Z"/>
> >      <dataset name="k85001182828.htn_d" serviceName="URI-DODS"
> >               urlPath="htn_sst_decloud/1985/k85002181750.htn_d.Z"/>
> >   </collection>
> > </catalog>

-- 
Ethan R. Davis                       Telephone: (303) 497-8155
Software Engineer                    Fax:       (303) 497-8690
UCAR Unidata Program Center          E-mail:    address@hidden
P.O. Box 3000
Boulder, CO  80307-3000              http://www.unidata.ucar.edu/

NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.