[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: announce stable release THREDDS Data Server version 3.4




Steve Hankin wrote:


John Caron wrote:

Thank you!

BTW, I'll be looking at your case next, in the "new types of aggregation".

Hi John,

I'll take that as an invitation to elaborate a little on the needs for forecast data set collections. To reiterate, the challenge is to expose three views of a collection of successive (operational) forecasts:

   1. the "integration foreward in time" axes (one virtual data set per
      model run)
   2. the "sequence of successive forecasts" axis (one virtual dataset
      per forecast time increment)
      and, if possible
   3. the "successively shorter forecast of the same endpoint" axes (one
      virtual dataset per end time)

just to make sure i understand, i am going to reword this:

a "ForecastModelRun" is a collection of all the forecasts for one model run. we 
make this into a single netcdf file.

a "ForecastModelRun collection" is a collection of ForecastModelRun files for 
the same model.

so
#1 = is a virtual dataset consisting of all the time=0 slices in each 
modelForecastRun, except the last, which uses all of the time steps.

#2 = is all of the time steps in a ForecastModelRun

#3 = the set of time steps with constant forecast time

Assuming this is correct, then we already have #2. #1 is probably 
straightforward to do as a special kind of aggregation on the server.

#3 is harder because there is a whole collection of these, one for each 
forecast time. Im not sure whether to make seperate datasets for each, or make 
a more complex object that is parameterized by forecast time (I assume thats 
what you mean by a 5-D dataset).

Any of these aggregatations could efficiently be done on a client, and i might 
be tempted to do it on the client in the nj22 library. But then only java apps 
could use it.

I will take your advice and investigate the vanilla server approach, ie making 
seperate (virtual) datasets on the server that can be accessed by any dods 
client.



The parentheses indicate where a special challenge is -- namely, that the collection proliferates into a number of discrete virtual datasets. This of course is a reflection of reality; each time the forecast model is run, it truly does generate a new dataset.

Here's what I want to add to the discussion: As a software developer -- striving for simplicity, compactness, elegance -- it may be tempting to solve this by using a more-than-4-dimensional array. I'd like to lobby that while of course it is fine (and fun) to explore new solution spaces, it is also vital to include in the solution an approach that makes the aggregations look like "standard" (CF) datasets -- i.e. 4-dimensional. Else, standard CF clients will not be able to read the datasets.

I suspect that this will involve something like the synthesis of a virtual "file system" in the URLs

    http://IP/server/forecast_collection ...
       /by_start_time...
          /2006-01-01
          /2006-01-02
          /2006-01-03
          ...
    /by_forecast_increment ...
         /z-0
         /z-4hrs
         /z-8hrs
         ...
    /by_end_time
         /2006-01-03
         /2006-01-04
         /2006-01-05
         ...

Just initial thoughts.  A unique problem, eh?

    - steve


Steve Hankin wrote:

Yahoo!  Congratulations.

   - steve

John Caron wrote:

A new stable release of the THREDDS Data Server 3.4 is now ready at

http://www.unidata.ucar.edu/projects/THREDDS/tech/index.html#TDS

Among other feaures, this version now has all the functionality of the older OpenDAP Aggregation Server (AS), which is no longer supported. The AS allowed aggregation of static sets of files, using "Union", "Join Existing" and "Join New" types. The TDS also adds automatic scans of directories, plus manipulation of the dataset metadata using NcML.

The configuration of the TDS is significantly different than the AS, since the TDS uses NcML Aggregation. See:

http://motherlode.ucar.edu:8080/thredds/docs/NcML.htm
http://www.unidata.ucar.edu/software/netcdf/ncml/

You are encouraged to upgrade to new stable releases as they become available, since various bugs are fixed. A stable release means that no further features will be added, and only bug fixes will be made. New development is now being done on the 3.5 version. This version will be adding more features, especially to deal with dynamically changing datasets, new types of aggregation, and some of the sophisticated features of TDS catalog generation (filter, sort, name) will be pushed into NcML aggregation. Use the development version only if you need the new features and are willing to bleed a little bit.





--
--

Steve Hankin, NOAA/PMEL -- address@hidden
7600 Sand Point Way NE, Seattle, WA 98115-0070
ph. (206) 526-6080, FAX (206) 526-6744