Re: [thredds] oceanotron, in-situ profiles through OPeNDAP interface

Hi Thomas:

Thanks.  Your timing couldn't have been better. There is a lot of discussion in 
NOAA right now on how to serve observational data.  I am certain this will be 
given a good look.

Regards,

-Roy

On Dec 7, 2010, at 9:34 AM, Thomas LOUBRIEU wrote:

> Dear Roy, dear all,
> 
> We've just published our in-situ data server at Ifremer. It is fully 
> compliant with what has been done with DAPPER (and Dchart can request our 
> interface).
> It disseminates vertical profiles datasets and soon it will disseminate 
> pointSeries (e.g. mooring datasets) and trajectories (e.g. drifting buoys or 
> ferry box).
> The URL  is : http://www.ifremer.fr/oceanotron/OPENDAP/
> 
> The request-able criteria are of course, x,y,z,t coordinates and id, but also 
> feature's metadata (platform code, institution) and phenomenons values (e.g. 
> temperatures). A session/pagination system is also implemented (with OPeNDAP 
> indices) to get big data subsets results with several requests each of them 
> not exceeding a quota.
> 
> The software is open-source and can be extended with modules to read datasets 
> (storage units), to transform datasets (transform unit) and disseminate the 
> datasets (front-desk). The communication between these modules, especially 
> observations features,  is standardized with Value Objects (their description 
> is available at : 
> http://www.ifremer.fr/isi/oceanotron/umlVOReport/oceanotron_uml.html)
> 
> You can get the software documentation and source from our wiki : 
> https://gforge.ifremer.fr/wiki/doku.php?id=oceanotron:start
> 
> If you have any question or comment, please do not hesitate to send it.
> 
> Best regards,
> 
> Thomas
> 
> On 12/06/2010 05:53 PM, Roy Mendelssohn wrote:
>> Agreed.  I said similar things at the meeting. But I was starting from the 
>> discussions at the meeting, which was the CDM as implemented in THREDDS.  So 
>> I want to get clear in my mind what TDS will be able to do for observational 
>> data first before looking for alternatives.
>> 
>> -Roy
>> 
>> On Dec 2, 2010, at 2:42 PM, Upendra Dadi wrote:
>> 
>>> Roy,
>>> With regard WODB, the users want to be able to search not just using 
>>> physical coordinates like lat-lon bounding box, but also using other 
>>> attributes like institution, project, platform etc which are usually not 
>>> represented using coordinate variables. So grouping the data based on 
>>> physical coordinates might cause performance issues when searching using 
>>> other criteria. Would it be possible to have some additional structures, 
>>> something like a database index, but without a full-fledge DBMS? Does TDS 
>>> allow such structures?
>>> 
>>> Upendra
>>> 
>>> 
>>> Roy Mendelssohn wrote:
>>>> Hi John:
>>>> 
>>>> Since the UAF meeting in Seattle I have been giving some thought about how 
>>>> to serve some large, important datasets, such as the raw ICOADS 
>>>> observations or the WODB observations. While reading over the 
>>>> PointObservation Conventions proposal on the CF site, while the proposal 
>>>> makes it clear how I might put data into a netcdf file, it doesn't make 
>>>> clear what the interplay might be with a service in TDS, and how a 
>>>> possible service might be affected by a very large dataset without further 
>>>> structure.
>>>> 
>>>> So it seems pretty clear that the ICOADS would be points.  From the 
>>>> example:
>>>> 
>>>> dimensions:
>>>>  obs = 1234 ;
>>>> 
>>>> variables:
>>>>  double time(obs) ;
>>>>    time:long_name = "time of measurement" ;
>>>>    time:units = "days since 1970-01-01 00:00:00" ;
>>>>  float lon(obs) ;
>>>>    lon:long_name = "longitude of the observation";
>>>>    lon:units = "degrees_east";
>>>>  float lat(obs) ;
>>>>    lat:long_name = "latitude of the observation" ;
>>>>    lat:units = "degrees_north" ;
>>>>  float alt(obs) ;
>>>>    alt:long_name = "vertical distance above the surface" ;
>>>>    alt:standard_name = "height" ;
>>>>    alt:units = "m";
>>>>    alt:positive = "up";
>>>>    alt:axis = "Z";
>>>> 
>>>>  float humidity(obs) ;
>>>>    humidity:long_name = "specific humidity" ;
>>>>    humidity:coordinates = "time lat lon alt" ;
>>>>  float temp(obs) ;
>>>>    temp:long_name = "temperature" ;
>>>>    temp:units = "Celsius" ;
>>>>    temp:coordinates = "time lat lon alt" ;
>>>> 
>>>> attributes:
>>>>  :CF\:featureType = "point";
>>>> Now I am assuming that in a TDS implementation of a service, I will be 
>>>> able to select on the coordinate variables, is that correct?  Even so, for 
>>>> something like ICOADS, obs is quite large and that extract could be quite 
>>>> slow unless either there is additional structure or the  TDS pre-fetches 
>>>> the coordinate variables much as the present Dapper server does.
>>>> 
>>>> Other options would be to say have a file for each 10-degree block, and 
>>>> then have TDS aggregate over the files - would this be possible.  Then the 
>>>> search would a lot faster when people want time series in a region as 
>>>> opposed to more synoptic extractions.  Would the TDS service be supporting 
>>>> such an option?  Or, as netcdf-4 supports groups, to have 10-degree groups 
>>>> with 2-degree subgroups, which would work as far as netcdf-4 is concerned, 
>>>> but that is not the same as TDS knowing what to do with the hierarchy or 
>>>> to take advantage of the structure.
>>>> 
>>>> My questions for Profiles  (that is for the WODB) are pretty much the 
>>>> same.  I assume that the TDS service will be able to search on the 
>>>> coordinate variables, is that correct?  And I have the issue with the fact 
>>>> that the profile dimension variable will get quite large and without 
>>>> further structure the search could be very slow.  Adding the same types of 
>>>> structures mentioned above would provide possible solutions, but only if 
>>>> TDS, as opposed to netcdf4, supported them.
>>>> 
>>>> As you may have guessed, these are not theoretical questions - I would 
>>>> really like to see ICOADS and WODB served as part of the year 2 UAF 
>>>> effort.  So now is a good time to start thinking about how to do it 
>>>> correctly and what the service will be able to do.
>>>> 
>>>> Thoughts?
>>>> 
>>>> Thanks,
>>>> 
>>>> -Roy
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> **********************
>>>> "The contents of this message do not reflect any position of the U.S. 
>>>> Government or NOAA."
>>>> **********************
>>>> Roy Mendelssohn
>>>> Supervisory Operations Research Analyst
>>>> NOAA/NMFS
>>>> Environmental Research Division
>>>> Southwest Fisheries Science Center
>>>> 1352 Lighthouse Avenue
>>>> Pacific Grove, CA 93950-2097
>>>> 
>>>> e-mail:Roy.Mendelssohn@xxxxxxxx  <mailto:Roy.Mendelssohn@xxxxxxxx>  (Note 
>>>> new e-mail address)
>>>> voice: (831)-648-9029
>>>> fax: (831)-648-8440
>>>> www:http://www.pfeg.noaa.gov/
>>>> 
>>>> "Old age and treachery will overcome youth and skill."
>>>> "From those who have been given much, much will be expected"
>>>> ------------------------------------------------------------------------
>>>> 
>>>> _______________________________________________
>>>> thredds mailing list
>>>> thredds@xxxxxxxxxxxxxxxx
>>>> For list information or to unsubscribe,  
>>>> visit:http://www.unidata.ucar.edu/mailing_lists/
>> **********************
>> "The contents of this message do not reflect any position of the U.S. 
>> Government or NOAA."
>> **********************
>> Roy Mendelssohn
>> Supervisory Operations Research Analyst
>> NOAA/NMFS
>> Environmental Research Division
>> Southwest Fisheries Science Center
>> 1352 Lighthouse Avenue
>> Pacific Grove, CA 93950-2097
>> 
>> e-mail:Roy.Mendelssohn@xxxxxxxx  (Note new e-mail address)
>> voice: (831)-648-9029
>> fax: (831)-648-8440
>> www:http://www.pfeg.noaa.gov/
>> 
>> "Old age and treachery will overcome youth and skill."
>> "From those who have been given much, much will be expected"
>> 
>> _______________________________________________
>> thredds mailing list
>> thredds@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe,  
>> visit:http://www.unidata.ucar.edu/mailing_lists/
> 
> -- 
> 
> 
> -------------------------------------------------------------
> Thomas LOUBRIEU
> IFREMER IDM/ISI
> BP70
> 29280 Plouzane
> FRANCE
> 
> email:Thomas.Loubrieu@xxxxxxxxxx
> WWW  :http://www.coriolis.eu.org/cdc
> Tel.:  (+33) (0)2 98 22 48 53
> Fax:   (+33) (0)2 98 22 46 44
> 
> -------------------------------------------------------------
> 
> 

**********************
"The contents of this message do not reflect any position of the U.S. 
Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected" 



  • 2010 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: