[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: datasetAggregation



Hi Bas,

Current aggregation capabilities have been targeted at particular types of homogeneous collections of datasets. In particular, the "Union" aggregation would work with a collection where different parameters were stored in different files but the dimensions of the arrays were the same. The "JoinExisting" aggregation would work with a collection containing all the same parameters in each dataset but the time dimension, for instance, changed from file to file. The "JoinNew" aggregation is similar to the "JoinExisting" but the variables wouldn't have a time dimension, the time dimension would be specified in the aggregation setup.

We haven't yet worked on any aggregation that would knit together varying geographic footprints.

Here is the documentation on NcML aggregation in general, it goes into some detail on "Union", "JoinExisting", and "JoinNew":

http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/Aggregation.html

Here is the documentation on using aggregation in the TDS:

http://motherlode.ucar.edu:8080/thredds/docs/NcML.htm

A side note:
Another possible future direction we sometimes talk about involves collection types and the searching/subsetting of collections. Kind of the flip side of aggregation or an alternate if aggregation seems too hard. Basically, it would involve providing a service that knows enough about a data collection to allow querying to retrieve the subset of the collection that matches your query. We haven't gone down this road as far as we have with aggregation but it might be a better fit for the situation where the geographic footprints of dataset don't mesh together easily.


Sorry none of this directly addresses your issue.

I'm going to CC one of the IDV guys and see if they have any thoughts on this.

Ethan


Bas Retsios wrote:
Hello Ethan,

It has been a while since our last communication (mainly due to the fact that it is holiday season).

I have made some improvements in CrawlableDatasetDods, to make it compatible with a few more DODS servers. When it is mature enough, I will submit it to you.

Meanwhile I am still learning what can be done with the catalog.xml files in the current Thredds version (3.10), because our original objective was to make the querying of MODIS images easier through Thredds.

I would like to ask some clarification for one of the aspects of Thredds, namely aggregation.

Check the following datasetScan section:

<datasetScan name="DodsTest2" path="MOD021KM.005" location="http://g0dup05u.ecs.nasa.gov/opendap-bin/nph-dods/OPENDAP_DP/short_term/MOGT/MOD021KM.005"; ID="nasaTest2" addDatasetSize="true" addLatest="true">
<crawlableDatasetImpl className="thredds.crawlabledataset.CrawlableDatasetDods" />
<metadata inherited="true">
<serviceName>remoteopendap2</serviceName>
</metadata>
</datasetScan>


This gives me a huge list of satellite images. Each image covers a different location on earth.

Is there a way to use the Thredds "aggregation" functionality in order to get one huge virtual image covering the entire earth? If yes, what elements should I add to my catalog.xml? (I found options datasetAggregation and netcdf\aggregation). Is there an example I can look at?

My purpose is to use IDV as a client on those images. The current behaviour (with the datasetScan above) is that IDV lists the images in the dataset, and I have to make a selection based on filename, without knowing the area covered. I would prefer indicating a bounding-box, and let IDV load the necessary images for me.

Thanks in advance,

Bas.




-- Ethan R. Davis Telephone: (303) 497-8155 Software Engineer Fax: (303) 497-8690 UCAR Unidata Program Center E-mail: address@hidden P.O. Box 3000 Boulder, CO 80307-3000 http://www.unidata.ucar.edu/ ---------------------------------------------------------------------------



NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.