Grads GDS / Re: GRIB Aggregation
John Caron
caron at unidata.ucar.edu
Fri Jul 7 17:13:01 MDT 2006
Sorry its taken so long to get back to you on this one.
One possibility:
NcML aggregation requires homogenous files, GRIB files are notorious for not being homogenous, because they are collections of 2D GRIB records. We are working on a more powerfil kind of aggregation called "Forecast Model Run" aggregation, that should solve this problem.
anyway, do you have error messages in the log files?
dan.swank wrote:
> TDS support :
>
> I have tried to get 3 seperate aggregations working during the few hours
> I've had to work on this during the past week.
>
> In our TDS test area, Please take a look at the following 3 entries:
> http://nomads.ncdc.noaa.gov:8085/thredds/testDatasets.html
>
> o Blended Ocean Winds Test Daily Aggregation
> o NARR-A Test Daily Aggregation
> o Test Narr hgtprs Subset
>
> The configuration file entries for these follow:
>
> ==============================================
>
> <!-- Test BOW Aggregation -->
>
> <dataset name="Blended Ocean Winds Test Daily Aggregation"
> ID="test/BOWdailyagg" urlPath="test/BOWdailyagg">
> <serviceName>allTest</serviceName>
> <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
>
> <dimension name="time" length="6688" />
>
> <variable name="time" type="int" shape="time">
> <attribute name="long_name" value="Aggregated Time axis in days" />
> <attribute name="units" value="days since 1987-07-09" />
> <attribute name="_CoordinateAxisType" value="Time" />
> <values start="0" incr="1" npts="6688" />
> </variable>
>
> <aggregation dimName="time" type="joinNew">
> <variableAgg name="wind" />
> <scan dateFormatMark="#yyyyMMdd"
> location="/eclipse1a/ftp/pub/seawinds/SI/daily/netcdf/1980s/"
> suffix=".nc" />
> <scan dateFormatMark="#yyyyMMdd"
> location="/eclipse1a/ftp/pub/seawinds/SI/daily/netcdf/1990s/"
> suffix=".nc" />
> <scan dateFormatMark="#yyyyMMdd"
> location="/eclipse1a/ftp/pub/seawinds/SI/daily/netcdf/2000s/"
> suffix=".nc" />
> </aggregation>
> </netcdf>
> </dataset>
>
> <!-- Test NARR Aggregation (local) -->
>
> <dataset name="NARR-A Test Daily Aggregation"
> ID="test/narragg" urlPath="test/narragg">
> <access serviceName="allTest" urlPath="test/narragg"
> dataFormat="GRIB-1" />
> <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
>
> <dimension name="time" length="8" />
> <variable name="time" type="int" shape="time">
> <attribute name="long_name" value="Aggregated Time axis in 3hr" />
> <attribute name="units" value="hours since 2006-01-01 00:00" />
> <attribute name="_CoordinateAxisType" value="Time" />
> <values start="0" incr="3" npts="8" />
> </variable>
>
> <aggregation dimName="time" type="joinNew">
> <variableAgg/>
> <scan dateFormatMark="#narr-a_211_yyyyMMdd_hhmm_000"
> location="/nomads3_data/raid5a/data/merged_AWIP32/200601/20060101"
> suffix=".grb" />
> </aggregation>
> </netcdf>
> </dataset>
>
> <!-- NARR hgtprs test agg (remote) -->
>
> <dataset
> name="Test Narr hgtprs Subset" ID="NARR/hgtprs"
> urlPath="NCEP_NARR_DAILY/narr-a_221_hgtprs.subset"
> serviceName="gds" >
> <catalogRef xlink:title="Test NARR Subset hgtprs"
>
> xlink:href="http://nomads.ncdc.noaa.gov:9091/dods/NCEP_NARR_DAILY/narr-a_221_hgtprs.subset.thredds"
> />
> <serviceName>gds</serviceName>
>
> <variable name="Geopotential Height" orgName="hgtprs">
> <attribute name="long_name" value="Geopotential Height on
> Pressure Levels" />
> <attribute name="units" value="gpm" />
> </variable>
> </dataset>
>
> ==============================================
> Services defined as follows:
> ==============================================
> <service name="allTest" serviceType="Compound" base="">
> <service name="ncdods" serviceType="OpenDAP" base="/thredds/dodsC/"/>
> <service name="HTTPServer" serviceType="HTTPServer"
> base="/thredds/fileServer/"/>
> <service name="WCS" serviceType="WCS" base="/thredds/wcs/"/>
> </service>
> <service name="gds" serviceType="DODS"
> base="http://nomads.ncdc.noaa.gov:9091/dods/" />
> ==============================================
>
> The Blended Ocean winds almost work,
> The OpENDAP interface comes up and allows the subsetting functions, but
> whenever I submit a request for anything that spans time steps it
> returns an error.
>
> The other two, I may not even have in the same ballpark.
> I thought it would be a good idea to just try a single day of NARR in an
> aggregation with a scan element, but even that ~ so far ~ gives nothing
> but a blank page for OpENDAP service.
>
> Roy and Jerome :
> I'd like to take up you offer to see your NcML configuration.
> So far I've had no luck with this on my own.
>
> Need some pointers on what I need in the configuration.
> Thanks in advance for any assistance.
>
> -Dan
>
>
>
> Roy Mendelssohn wrote the following on 6/15/2006 10:44 AM:
>
>>TDS in fact allows for just what you are asking ('If it is possible to
>>create hooks that allow TDS to read and import data/metadata from a
>>source OPeNDAP URL') and we have a test of this implemented (go to
>>http://oceanwatch.pfeg.noaa.gov:8081/thredds/catalog.html and look at
>>'Test remote AUV data TDS'. These data are served by OPeNDAP at MBARI,
>>but we both aggregate and serve them through TDS at our sie.
>>
>>If you want info on how the THREDDS catalogs and NcML files are set up
>>to do this, email Jeorme King at jerome.king at noaa.gov. I would
>>apprecaiae being cc:'ed ; also.
>>
>>Hope this helps.
>>
>>-Roy
>>
>>
>>At 10:07 AM -0400 6/15/06, dan.swank wrote:
>>
>>
>>>We have had the NARR on GDS for quite some time :
>>>http://nomads.ncdc.noaa.gov:9091/dods/NCEP_NARR_DAILY
>>>GDS works very well with GRIB data in general.
>>>Our particular goal is to get it on TDS due to its suporior metadata
>>>handling and WCS service.
>>
>>>If it is possible to create hooks that allow TDS to read and import
>>
>>>data/metadata from a source OPeNDAP URL, similar to what the Live Access
>>>Server does... that would be outstanding. May pose an easier task to
>>>implement as well.
>>
>>>-Dan
>>
>>>
>>>Rob Cermak wrote the following on 6/14/2006 6:50 PM:
>>>
>>>>Dan,
>>>>
>>>>The Grads GDS OPeNDAP server is suppose to work with GRIB files. The
>>
>>>>large unknown is if TDS can then aggregate the data across the Grads GDS
>>>>server. To help answer that is to see if TDS can aggregate across maybe
>>>>one of the NOMADs GFS servers? I'm pretty sure they are using the Grads
>>>>GDS server.
>>>>
>>>>http://www.iges.org/grads/gds/
>>>>
>>>>Rob
>>>>
>>>>On Tue, June 13, 2006 2:28 pm, dan.swank wrote:
>>>>
>>>>
>>>>>Ethan:
>>>>>Understood. consider this a wish-list then :)
>>>>>-Dan
>>>>>
>>>>>Ethan Davis wrote the following on 6/13/2006 6:22 PM:
>>>>>
>>>>>
>>>>>>Hi Dan,
>>>>>>
>>>>>>Evidently, aggregation of GRIB data may not be working. I thought since
>>>>>>GRIB files are read by the netCDF-java library it would just work but I
>>>>>>just heard that there may be some issues with aggregating GRIB
>>>
>>>involving
>>>
>>>>>>how the underlying code writes indexes into the GRIB files. We'll have
>>>>>>to wait till John is back from vacation for a more detailed
>>>>>>answer/explanation.
>>>>>>
>>>>>>Sorry for the mistake.
>>>>>>
>>>>>>There currently isn't a way to have a single scan element scan a set of
>>>>>>directories.
>>>>>>
>>>>>>I believe the underlying netCDF-java code that reads GRIB files should
>>>>>>result in fairly well-formed, CF (I think) compliant netCDF views
>>>
>>>of the
>>>
>>>>>>dataset. Not sure how this would be affected by aggregation.
>>>>>>
>>>>>>Ethan
>>>>>>
>>>>>>dan.swank wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>>Ethan:
>>>>>>>
>>>>>>>The NARR is a reanalysis, so it don't have forecast times. I would be
>>>>>>>a
>>>>>>>simple 03 hr chain (00 hr fct time) spanning 26 years.
>>>>>>>
>>>>>>>See an existing GDS subset aggregation:
>>>>>>>http://nomads.ncdc.noaa.gov:9091/dods/NCEP_NARR_DAILY/narr-a_221_tmpprs.subset.info
>>>>>>>
>>>>>>>This will give a sense for the nature of the beast.
>>>>>>>
>>>>>>>The directory structure is set up as such:
>>>>>>>http://nomads.ncdc.noaa.gov/data/narr/
>>>>>>>
>>>>>>>
>>>>>>>Heres the TDS aggregation I set up while experimenting yesterday, on a
>>>>>>>non-related dataset:
>>>>>>>
>>>>>>> <dataset name="OceanWinds Test Daily Aggregation"
>>>>>>> ID="test/dailyagg" urlPath="test/agg">
>>>>>>> <serviceName>allTest</serviceName>
>>>>>>> <netcdf
>>>>>>>xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
>>>>>>> <aggregation dimName="time" type="joinNew">
>>>>>>> <variableAgg name="wind" />
>>>>>>> <scan dateFormatMark="#yyyyMMdd"
>>>>>>>location="/eclipse1a/ftp/pub/seawinds/SI/daily/netcdf/1980s/"
>>>>>>>suffix=".nc" />
>>>>>>> <scan dateFormatMark="#yyyyMMdd"
>>>>>>>location="/eclipse1a/ftp/pub/seawinds/SI/daily/netcdf/1990s/"
>>>>>>>suffix=".nc" />
>>>>>>> <scan dateFormatMark="#yyyyMMdd"
>>>>>>>location="/eclipse1a/ftp/pub/seawinds/SI/daily/netcdf/2000s/"
>>>>>>>suffix=".nc" />
>>>>>>> </aggregation>
>>>>>>> <variable name="time" orgName="time">
>>>>>>> <attribute name="long_name" value="Days"/>
>>
>>>>>>> <attribute name="units" value="days since 1987-07-09" />
>>>>>>> </variable>
>>>>>>> </netcdf>
>>>>>>> </dataset>
>>>>>>>
>>>>>>>
>>>>>>>Would this automatically detect the source of data were GRIB rather
>>>>>>>than
>>>>>>>NetCDF? and it seems like you need to set the <scan> on each
>>>>>>>individual
>>>>>>>directory... Doing so the way NARR is set up would create one chunky
>>>>>>>configuration file. Is there anyway to have this scan a pattern
>>>>>>>(YYYYMM/YYYYMMDD) of directories?
>>>>>>>
>>>>>>>I understand GRIB requires a certain amount of "supplemented" metadata
>>>>>>>for complience. Where do you enter this?
>>>>>>>
>>>>>>>-Dan
>>>>>>>
>>>>>>>
>>>>>>>Ethan Davis wrote the following on 6/13/2006 1:21 PM:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>Hi Dan,
>>>>>>>>
>>>>>>>>Aggregation should work the same for GRIB as for netCDF files. The
>>>>>>>>issue
>>>>>>>>would be how your GRIB files are structured and how you want to
>>>>>>>>aggregate them. Our GRIB files each contain one full model run (all
>>>>>>>>parameters, all forecast times). We haven't tried aggregating beyond
>>>>>>>>that.
>>>>>>>>
>>>>>>>>We have started tracking what is available for the NCEP models on our
>>>>>>>>server. This is from the TDS 3.8 announcement (with links updated):
>>>>>>>>
>>>>>>>> We also are now tracking detailed inventory of NCEP model output,
>>>>>>>>eg:
>>>>>>>>
>>>>>>>>http://motherlode.ucar.edu:8080/thredds/modelInventory/model/NCEP/NAM/CONUS_12km/
>>>>>>>>
>>>>>>>>
>>>>>>>> These are all linked from the "collection dataset" pages; For
>>>>>>>> example from
>>>>>>>>
>>>>>>>>http://motherlode.ucar.edu:8080/thredds/catalog/model/NCEP/NAM/CONUS_12km/catalog.html
>>>>>>>>
>>>>>>>>
>>>>>>>> choose the top "CONUS_12_km" link, then choose "Available
>>>>>>>>Inventory"
>>>>>>>> Documentation.
>>>>>>>>
>>>>>>>>One idea for this work is to eventually provide access to alternate
>>>>>>>>datasets, for instance, a dataset that contains all the 3hr forecast
>>>>>>>>times from the different runs, or one that contained all the 12Z
>>>
>>>valid
>>>
>>>>>>>>times from the different runs. Tracking these detailed inventories is
>>>>>>>>just the first step but aggregation and alternate groupings of the
>>>>>>>>data
>>>>>>>>is pretty interesting to think about.
>>>>>>>>
>>>>>>>>How are your GRIB files structured and what kind of aggregation where
>>>>>>>>you thinking about?
>>>>>>>>
>>>>>>>>Ethan
>>>>>>>>
>>>>>>>>dan.swank wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>Hello,
>>>>>>>>>
>>>>>>>>>I've been tinkering with the TDS aggregation capabilities and they
>>>>>>>>>work
>>>>>>>>>quite well for NetCDF data, however, I can't seem to find
>>>
>>>anything in
>>>
>>>>>>>>>the docs regarding aggregating GRIB.
>>>>>>>>>We want to get The NARR dataset which we have here at NCDC-NOMADS on
>>>>>>>>>the
>>>>>>>>>TDS. It consists of hundreds of thousands of 50 Mb + GRIB files
>>>
>>>in a
>>>
>>>>>>>>>YYYYMM/YYYYMMDD tree.
>>>>>>>>> Just scouting for a quick answer here:
>>>>>>>>>Is aggregating the NARR GRIB currently feasable with the current
>>>>>>>>>release
>>>>>>>>>of TDS? If so, do any docs exist which could give me a starting
>>>>>>>>>point?
>>>>>>>>>Converting it to NetCDF will not be possible (volume).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>--
>>>>>Dan Swank <dan.swank at noaa.gov>
>>>>>NOMADS Project: Software & Data Management
>>>>>Contractor - STG, Incorporated
>>>>>Veach-Baley Federal Building
>>>>>151 Patton Avenue
>>>>>Asheville, NC 28801-5001
>>>>>Phone: 828-271-4007
>>>>>
>>>>>==============================================================================
>>>>>To unsubscribe thredds, visit:
>>>>>http://www.unidata.ucar.edu/mailing-list-delete-form.html
>>>>>==============================================================================
>>>>>
>>>>
>>>>
>>>>
>>>--
>>>Dan Swank <dan.swank at noaa.gov>
>>>NOMADS Project: Software & Data Management
>>>Contractor - STG, Incorporated
>>>Veach-Baley Federal Building
>>>151 Patton Avenue
>>>Asheville, NC 28801-5001
>>>Phone: 828-271-4007
>>>
>>>==============================================================================
>>>To unsubscribe thredds, visit:
>>>http://www.unidata.ucar.edu/mailing-list-delete-form.html
>>>==============================================================================
>>
>>
>>
>>--
>>
>>**********************
>>"The contents of this message do not reflect any position of the U.S.
>>Government or NOAA."
>>**********************
>>Roy Mendelssohn
>>Supervisory Operations Research Analyst
>>NOAA/NMFS
>>Environmental Research Division
>>Southwest Fisheries Science Center
>>1352 Lighthouse Avenue
>>Pacific Grove, CA 93950-2097
>>
>>e-mail: Roy.Mendelssohn at noaa.gov (Note new e-mail address)
>>voice: (831)-648-9029
>>fax: (831)-648-8440
>>www: http://www.pfeg.noaa.gov/
>>
>>"Old age and treachery will overcome youth and skill."
>
>
==============================================================================
To unsubscribe thredds, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html
==============================================================================
More information about the Thredds
mailing list