Re: [thredds] TDS, CDM, Points and Profiles and large datasets

Agreed.  I said similar things at the meeting. But I was starting from the 
discussions at the meeting, which was the CDM as implemented in THREDDS.  So I 
want to get clear in my mind what TDS will be able to do for observational data 
first before looking for alternatives.

-Roy

On Dec 2, 2010, at 2:42 PM, Upendra Dadi wrote:

> Roy,
> With regard WODB, the users want to be able to search not just using physical 
> coordinates like lat-lon bounding box, but also using other attributes like 
> institution, project, platform etc which are usually not represented using 
> coordinate variables. So grouping the data based on physical coordinates 
> might cause performance issues when searching using other criteria. Would it 
> be possible to have some additional structures, something like a database 
> index, but without a full-fledge DBMS? Does TDS allow such structures?
> 
> Upendra
> 
> 
> Roy Mendelssohn wrote:
>> Hi John:
>> 
>> Since the UAF meeting in Seattle I have been giving some thought about how 
>> to serve some large, important datasets, such as the raw ICOADS observations 
>> or the WODB observations. While reading over the PointObservation 
>> Conventions proposal on the CF site, while the proposal makes it clear how I 
>> might put data into a netcdf file, it doesn't make clear what the interplay 
>> might be with a service in TDS, and how a possible service might be affected 
>> by a very large dataset without further structure.
>> 
>> So it seems pretty clear that the ICOADS would be points.  From the example:
>> 
>> dimensions:
>>  obs = 1234 ;
>> 
>> variables:
>>  double time(obs) ;
>>    time:long_name = "time of measurement" ;
>>    time:units = "days since 1970-01-01 00:00:00" ;
>>  float lon(obs) ;
>>    lon:long_name = "longitude of the observation";
>>    lon:units = "degrees_east";
>>  float lat(obs) ;
>>    lat:long_name = "latitude of the observation" ;
>>    lat:units = "degrees_north" ;
>>  float alt(obs) ;
>>    alt:long_name = "vertical distance above the surface" ;
>>    alt:standard_name = "height" ;
>>    alt:units = "m";
>>    alt:positive = "up";
>>    alt:axis = "Z";
>> 
>>  float humidity(obs) ;
>>    humidity:long_name = "specific humidity" ;
>>    humidity:coordinates = "time lat lon alt" ;
>>  float temp(obs) ;
>>    temp:long_name = "temperature" ;
>>    temp:units = "Celsius" ;
>>    temp:coordinates = "time lat lon alt" ;
>> 
>> attributes:
>>  :CF\:featureType = "point";
>> Now I am assuming that in a TDS implementation of a service, I will be able 
>> to select on the coordinate variables, is that correct?  Even so, for 
>> something like ICOADS, obs is quite large and that extract could be quite 
>> slow unless either there is additional structure or the  TDS pre-fetches the 
>> coordinate variables much as the present Dapper server does.
>> 
>> Other options would be to say have a file for each 10-degree block, and then 
>> have TDS aggregate over the files - would this be possible.  Then the search 
>> would a lot faster when people want time series in a region as opposed to 
>> more synoptic extractions.  Would the TDS service be supporting such an 
>> option?  Or, as netcdf-4 supports groups, to have 10-degree groups with 
>> 2-degree subgroups, which would work as far as netcdf-4 is concerned, but 
>> that is not the same as TDS knowing what to do with the hierarchy or to take 
>> advantage of the structure.
>> 
>> My questions for Profiles  (that is for the WODB) are pretty much the same.  
>> I assume that the TDS service will be able to search on the coordinate 
>> variables, is that correct?  And I have the issue with the fact that the 
>> profile dimension variable will get quite large and without further 
>> structure the search could be very slow.  Adding the same types of 
>> structures mentioned above would provide possible solutions, but only if 
>> TDS, as opposed to netcdf4, supported them.
>> 
>> As you may have guessed, these are not theoretical questions - I would 
>> really like to see ICOADS and WODB served as part of the year 2 UAF effort.  
>> So now is a good time to start thinking about how to do it correctly and 
>> what the service will be able to do.
>> 
>> Thoughts?
>> 
>> Thanks,
>> 
>> -Roy
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> **********************
>> "The contents of this message do not reflect any position of the U.S. 
>> Government or NOAA."
>> **********************
>> Roy Mendelssohn
>> Supervisory Operations Research Analyst
>> NOAA/NMFS
>> Environmental Research Division
>> Southwest Fisheries Science Center
>> 1352 Lighthouse Avenue
>> Pacific Grove, CA 93950-2097
>> 
>> e-mail: Roy.Mendelssohn@xxxxxxxx <mailto:Roy.Mendelssohn@xxxxxxxx> (Note new 
>> e-mail address)
>> voice: (831)-648-9029
>> fax: (831)-648-8440
>> www: http://www.pfeg.noaa.gov/
>> 
>> "Old age and treachery will overcome youth and skill."
>> "From those who have been given much, much will be expected" 
>> ------------------------------------------------------------------------
>> 
>> _______________________________________________
>> thredds mailing list
>> thredds@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe,  visit: 
>> http://www.unidata.ucar.edu/mailing_lists/ 
> 

**********************
"The contents of this message do not reflect any position of the U.S. 
Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected" 



  • 2010 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: