Re: THREDDS aggregation capability

[Ben Burford <benb@xxxxxxxxxxxx>]

Hello Roy, Jose, Ethan and All,

Thanks very much for your help.

The project that I'm working on is CEOP (http://www.ceop.net/) and we are re
ceiving this data from 10 major data centers internationally (NCEP, UK Met O
ffice, ECPC, ECMWF, BMRC, JMA, Epson Meteo Centre (EMC), GLDAS, GMAO and CPT
EC).  Right now I want to focus on 2, NCEP and UK Met Office (UKMO).  I thin
k I can get the NCEP data in netCDF and I'll send more information on this l
ater.

It turns out that the UKMO data is not in NetCDF and it may be a while befor
e we have the tool for converting the ascii data to netCDF, so I would like 
to ask you to tell me if we can work with the UKMO data in ascii.

A sample file is attached along with some documentation on the format of the
 data.  Remember, this is time series data at a grid point (derived from glo
bal gridded NWP model output data).

There is one file per day, with a 36 hour forecast beginning at 12Z.  This 3
6 hour forecast produced values on a 3 hourly interval, so, in each file the
re are 13 data values per variable (i.e. 12Z+00  12Z+03  12Z+06  12Z+09  12Z
+12  12Z+15  12Z+18  12Z+21  12Z+24  12Z+27  12Z+30  12Z+33  12Z+36).  The d
ata is space separated.

Following is an example of a file name: UKMO_ST_FC_12Z_lin_MUL_1.txt.  As yo
u can see the date/time is not given in the file name.  This file begins at:
 20021001  12Z.  The final digit in the file name (in this case its 1) gives
 the record number, so this digit would go from 1 to 365 for the first year 
of data.

There is a detailed description of the format in the attached file, but I'll
 summarize the format as follows:
1. Header information (date/time/lat/lon/elevation/etc.)

2. Values for surface variables
This section contains the 13 data values for 41 variables at a single level 
(e.g. surface) (referred to as "single level data" in the file).  Each line 
contains some information (e.g. variable name) followed by 13 data values.  
The following example is one line of 13 time series data values for TotSW do
wn TOA.

128 TotSW down TOA            W/m2     3        7.445994e+02  5.875879e+02  
1.083438e+02  0.000000e+00  0.000000e+00  0.000000e+00  1.670333e+01  4.2427
90e+02  7.375867e+02  5.797017e+02  1.035093e+02  0.000000e+00  0.000000e+00


3. Values for variables at pressure levels
This section contains the 13 time series data values (per pressure level) fo
r 8 variables at 18 pressure levels.  Each line contains the pressure level 
followed by 13 time series data values at one pressure level.  The following
 example is one line of data for 13 times of temperature data values at pres
sure =1000:

1000.0       2.891250e+02  2.896250e+02  2.895000e+02  2.890000e+02  2.88375
0e+02  2.875000e+02  2.866250e+02  2.860000e+02  2.895000e+02  2.907500e+02 
 2.903750e+02  2.892500e+02  2.876250e+02


4. Values for variables at model levels (height in meters)
This sections contains the 13 time series data values (per model level) for 
the 11 variables at 38 model levels.  Each line contains the level number (1
 to 38), the height in meters, followed by the 13 time series data values fo
r one variable at one model level.  The following example is one line of dat
a for 13 times of temperature data values at model level 1, height of 19.9 m
eters:

  1     19.9  2.903750e+02  2.908750e+02  2.907500e+02  2.890000e+02  2.8662
50e+02  2.840000e+02  2.818750e+02  2.852500e+02  2.908750e+02  2.920000e+02
  2.908750e+02  2.871250e+02  2.851250e+02


Any possibility of putting this data on a THREDDS (or OPeNDAP, or whatever) 
server and then being able to extract a subset of the data (e.g. 12Z+15 to 1
2Z+36) from each file and then concatenating this into a continuous time ser
ies?  If its not possible to do the variables at pressure or model levels, w
ould it be possible to at least do this for the variables at "single level d
ata"?

Thanks again for your help.

Ben



At 13:52 2006/06/16 -0600, you wrote:
> Hi Ben, all,
> 
> This conversation got split between dods-tech and thredds. Sorry for the 
> duplicate cross-posting but I'm going to try to merge this back into one 
> conversation. So, a post from Jose and one from Roy follow.
> 
> Also, we might be better able to advise if we knew more about the format 
> and structure of the data you want to serve. An "ncdump -h" if netCDF 
> files. Or a pointer to an example file.
> 
> Ethan
> 
> Jose Garcia wrote:
> > Hi,
> >
> > Assuming your data is in netCDF format, I am happy to say that we have 
> > a very similar requirement for a project called the Earth System Grid. 
> > For this, we have developed an aggregation module that plugs into the 
> > OPeNDAP server. This module requires OPeNDAP server version 4, which 
> > is I understand is not the latest release of OPeNDAP,  
> > but nevertheless can be used now.
> >
> > If you wish to take the steps to build the system on your institution, 
> > we will be glad to support the process. 
> >
> > Please let us know,
> >
> > Jose Garcia
> > NCAR/HAO
> > Boulder, CO
> > USA
> 
> Roy Mendelssohn wrote:
>  > Hi Ben:
>  >
>  > THREDDS allows you to aggregate along any dimension, including a 
> "new" one.  In your case, you would aggregate across time, and then do 
> the normal OPeNDAP call for the one location, treating it as if it were 
> one series.
>  >
>  > You can see a lot of examples of this at:
>  >
>  > http://oceanwatch.pfeg.noaa.gov:8081/thredds/catalog.html
>  >
>  > All of the Items under the heading  "Aggregation Satellite Datasets" 
> are aggregated in just this fashion . You can pick one and go to the 
> html interface and see that you can select to get a time series at a 
> point.  In fact, you can get a time series for a region.
>  >
>  > HTH,
>  >
>  > -Roy M.
>  >
>  >> I'm wondering if THREDDS (or perhaps simply a DODS or other type of 
> server)
>  >> can help me with the following data aggregation problem.
>  >>
>  >> I have daily data files containing an 84 hour forecast.  From these 
> daily fi
>  >> les I want to extract a 24 hour section out of the 84 hour forecast 
> and conc
>  >> atenate those 24 hour sections to form a continuous time series.  
> There may
>  >> be multiple sections of interest (e.g. 6 to 30, 36 to 50, 60 to 
> 84).  This i
>  >> s "station" time series data at a point.  It is actually the time 
> series at
>  >> a grid point extracted from daily NWP data files.
>  >>
>  >> Can THREDDS do this?  Or is there another type of server you know of 
> that wi
>  >> ll handle this problem?
>  >>
>  >> Thank you,
>  >>
>  >> Ben
> 
> -- 
> Ethan R. Davis                                Telephone: (303) 497-8155
> Software Engineer                             Fax:       (303) 497-8690
> UCAR Unidata Program Center                   E-mail:    edavis@xxxxxxxx
> P.O. Box 3000
> Boulder, CO  80307-3000                       http://www.unidata.ucar.edu/
> ---------------------------------------------------------------------------
>

Attachment: UKMO_ST_FC_12Z_lin_MUL_1.txt
Description: Binary data

Attachment: UKMO_notes_on_data.doc
Description: MS-Word document