[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Introductory THREDDS question

-- Ethan R. Davis Telephone: (303) 497-8155 Software Engineer Fax: (303) 497-8690 UCAR Unidata Program Center E-mail: address@hidden P.O. Box 3000 Boulder, CO 80307-3000 http://www.unidata.ucar.edu/ ---------------------------------------------------------------------------

--- Begin Message ---
  • To: Peter Mooney <address@hidden>
  • Subject: Re: Introductory THREDDS question
  • From: Ethan Davis <address@hidden>
  • Date: Wed, 21 Dec 2005 10:43:20 -0700
Hi Peter,

The THREDDS catalog XML was designed to be used to describe and provide access information for pretty much any data sets. The access information for a dataset is a URL and an access/service type. The metadata that can be included in a catalog is very flexible. We have defined a basic level of metadata but, also, any XML can be contained in or a URL can be referenced by the <metadata> element.

The THREDDS Data Server (TDS) can serve data via HTTP and OPeNDAP and does automatic catalog generation for the data it is serving. It can also catalog data served by other servers. It can serve any type of data over HTTP. For a dataset to be served via OPeNDAP, it must be readable by the netCDF-Java library. The netCDF-Java library (ver 2.2) can currently read netCDF-3 files, HDF5, GRIB, GINI, NEXRAD radar data (level 2 and 3), as well as data served by OPeNDAP and ADDE servers.

If you have other types of data and want them served via OPeNDAP, the netCDF-Java library does provide an interface that can be implemented for new data formats allowing them to be read by the netCDF-Java library.

The TDS does not come with any large scale or cross-catalog search/discovery service. We do use our DQC documents in a number of places. We mainly use DQCs (Dataset Query Capability) for requesting a subset of a homogeneous dataset collection, for instance, the last hour of radar data from a certain station. We have been thinking some about extending the DQC for more extensive discovery but that is still in the thinking stage. We work pretty closely with a group that crawls a network of THREDDS catalogs to gather up metadata information and provide a search service on that collection of datasets. We are also working with some other discovery service providers to feed them our metadata records.

Hope that helps. Let me know if you have any other questions.


Peter Mooney wrote:

 Hello there,

My name is Peter Mooney and I am currently working with the Environmental Protection
Agency in Ireland. I am working on IT Systems Development. I subscribe to the THREDDS mailing list
and am currently begining to dabble in the world of THREDDS and netCDF. I apologise for sending an email directly to you guys but I don't want to send it to the entire list as it may be off the point of the THREDDS discussion.

A small overview of background to this project.
EPA-Ireland invests significantly in research programmes in MSc and PhD programmes in
universities and colleges in Ireland. In the past the data and results that were
generated from these research projects were effectively lost in time - the results were
published in journals but the data was not archived. In my work group we have
build a web-based system to allow funded projects to "upload" data (by FTP and HTTP) to our
SAN server for safe archive and storage of these datasets and results. The EPA-Ireland
have committed to safe-guarding this information going forward. Many of the datasets
generated are of interested to other researchers, stakeholders, the general public etc.
As you will agree this is where the real problems start - how to standardise datasets
coming in from many different projects/systems/locations. This is compounded by the fact that
they have to be archived and made available to allow as wide a range of stakeholders to
access them.

We have now begun to look at providing web services such as THREDDS.
We are working with Tomcat/Apache/Servlets etc in our everyday work so we are comfortable
with working with this excellent system.
We have a few questions specific to our work in relation to THREDDS - I wonder could
I be so bold as to ask them? I understand how busy you guys are and with the holiday season
approaching and all. I would be very grateful of any information.

We are considering a process of converting datasets we have in our archive to netCDF
and then using THREDDS to provide these datasets in a Web Service architecture.
However given the variety and 'quirkiness' of some of the data this could be a difficult
task. Is it possible to use THREDDS such that it can provide a Web Service to datasets
not in netCDF format? Could we use it to index our datasets in their original format?
We would like users to be able to get listings of our data holdings and then retrieve datasets
that match their specific queries (queries based on the metadata associated with each data resource) The team I work in (3 in total) are not concerned with having to roll up the sleves and do some
software dev work to make THREDDS work in the way described above.

Apologies if I have not articulated my ideas in a coherent manner - I would be happy to re-write my
ideas if you should require it.

I look forward to your reply,

Thank you again for your time,

Best wishes and have a nice day,

Peter Mooney

Play Santa's Celebrity Xmas Party <http://santas-christmas-party.yahoo.net/>, an exclusive game from Yahoo!

Ethan R. Davis                                Telephone: (303) 497-8155
Software Engineer                             Fax:       (303) 497-8690
UCAR Unidata Program Center                   E-mail:    address@hidden
P.O. Box 3000
Boulder, CO  80307-3000                       http://www.unidata.ucar.edu/

--- End Message ---

NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.