Grads GDS / Re: GRIB Aggregation

John Caron caron at unidata.ucar.edu
Thu Jun 22 17:46:02 MDT 2006 

Hi Dan:



dan.swank wrote:
> We have had the NARR on GDS for quite some time :
> http://nomads.ncdc.noaa.gov:9091/dods/NCEP_NARR_DAILY
> GDS works very well with GRIB data in general.
> Our particular goal is to get it on TDS due to its suporior metadata
> handling and WCS service.
> 
> If it is possible to create hooks that allow TDS to read and import
> data/metadata from a source OPeNDAP URL, similar to what the Live Access
> Server does... that would be outstanding.  May pose an easier task to
> implement as well.

this is on our list to do. Can you give us a concrete example of what you would like to see put into the thredds metadata?



> 
> -Dan
> 
> 
> Rob Cermak wrote the following on 6/14/2006 6:50 PM:
> 
>>Dan,
>>
>>The Grads GDS OPeNDAP server is suppose to work with GRIB files.  The
>>large unknown is if TDS can then aggregate the data across the Grads GDS
>>server.  To help answer that is to see if TDS can aggregate across maybe
>>one of the NOMADs GFS servers?  I'm pretty sure they are using the Grads
>>GDS server.
>>
>>http://www.iges.org/grads/gds/
>>
>>Rob
>>
>>On Tue, June 13, 2006 2:28 pm, dan.swank wrote:
>>
>>
>>>Ethan:
>>>Understood.  consider this a wish-list then :)
>>>-Dan
>>>
>>>Ethan Davis wrote the following on 6/13/2006 6:22 PM:
>>>
>>>
>>>>Hi Dan,
>>>>
>>>>Evidently, aggregation of GRIB data may not be working. I thought since
>>>>GRIB files are read by the netCDF-java library it would just work but I
>>>>just heard that there may be some issues with aggregating GRIB involving
>>>>how the underlying code writes indexes into the GRIB files. We'll have
>>>>to wait till John is back from vacation for a more detailed
>>>>answer/explanation.
>>>>
>>>>Sorry for the mistake.
>>>>
>>>>There currently isn't a way to have a single scan element scan a set of
>>>>directories.
>>>>
>>>>I believe the underlying netCDF-java code that reads GRIB files should
>>>>result in fairly well-formed, CF (I think) compliant netCDF views of the
>>>>dataset. Not sure how this would be affected by aggregation.
>>>>
>>>>Ethan
>>>>
>>>>dan.swank wrote:
>>>>
>>>>
>>>>
>>>>>Ethan:
>>>>>
>>>>>The NARR is a reanalysis, so it don't have forecast times.  I would be
>>>>>a
>>>>>simple 03 hr chain (00 hr fct time) spanning 26 years.
>>>>>
>>>>>See an existing GDS subset aggregation:
>>>>>http://nomads.ncdc.noaa.gov:9091/dods/NCEP_NARR_DAILY/narr-a_221_tmpprs.subset.info
>>>>>
>>>>>This will give a sense for the nature of the beast.
>>>>>
>>>>>The directory structure is set up as such:
>>>>>http://nomads.ncdc.noaa.gov/data/narr/
>>>>>
>>>>>
>>>>>Heres the TDS aggregation I set up while experimenting yesterday, on a
>>>>>non-related dataset:
>>>>>
>>>>> <dataset name="OceanWinds Test Daily Aggregation"
>>>>>     ID="test/dailyagg" urlPath="test/agg">
>>>>>   <serviceName>allTest</serviceName>
>>>>>   <netcdf
>>>>>xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
>>>>>    <aggregation dimName="time" type="joinNew">
>>>>>      <variableAgg name="wind" />
>>>>>      <scan dateFormatMark="#yyyyMMdd"
>>>>>location="/eclipse1a/ftp/pub/seawinds/SI/daily/netcdf/1980s/"
>>>>>suffix=".nc" />
>>>>>      <scan dateFormatMark="#yyyyMMdd"
>>>>>location="/eclipse1a/ftp/pub/seawinds/SI/daily/netcdf/1990s/"
>>>>>suffix=".nc" />
>>>>>      <scan dateFormatMark="#yyyyMMdd"
>>>>>location="/eclipse1a/ftp/pub/seawinds/SI/daily/netcdf/2000s/"
>>>>>suffix=".nc" />
>>>>>    </aggregation>
>>>>>    <variable name="time" orgName="time">
>>>>>      <attribute name="long_name" value="Days"/>
>>>>>      <attribute name="units" value="days since 1987-07-09" />
>>>>>    </variable>
>>>>>   </netcdf>
>>>>> </dataset>
>>>>>
>>>>>
>>>>>Would this automatically detect the source of data were GRIB rather
>>>>>than
>>>>>NetCDF?  and it seems like you need to set the <scan> on each
>>>>>individual
>>>>>directory... Doing so the way NARR is set up would create one chunky
>>>>>configuration file.  Is there anyway to have this scan a pattern
>>>>>(YYYYMM/YYYYMMDD) of directories?
>>>>>
>>>>>I understand GRIB requires a certain amount of "supplemented" metadata
>>>>>for complience.  Where do you enter this?
>>>>>
>>>>>-Dan
>>>>>
>>>>>
>>>>>Ethan Davis wrote the following on 6/13/2006 1:21 PM:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>Hi Dan,
>>>>>>
>>>>>>Aggregation should work the same for GRIB as for netCDF files. The
>>>>>>issue
>>>>>>would be how your GRIB files are structured and how you want to
>>>>>>aggregate them. Our GRIB files each contain one full model run (all
>>>>>>parameters, all forecast times). We haven't tried aggregating beyond
>>>>>>that.
>>>>>>
>>>>>>We have started tracking what is available for the NCEP models on our
>>>>>>server. This is from the TDS 3.8 announcement (with links updated):
>>>>>>
>>>>>>  We also are now tracking detailed inventory of NCEP model output,
>>>>>>eg:
>>>>>>
>>>>>>http://motherlode.ucar.edu:8080/thredds/modelInventory/model/NCEP/NAM/CONUS_12km/
>>>>>>
>>>>>>
>>>>>>  These are all linked from the "collection dataset" pages; For
>>>>>>  example from
>>>>>>
>>>>>>http://motherlode.ucar.edu:8080/thredds/catalog/model/NCEP/NAM/CONUS_12km/catalog.html
>>>>>>
>>>>>>
>>>>>>  choose the top "CONUS_12_km" link, then choose "Available
>>>>>>Inventory"
>>>>>>  Documentation.
>>>>>>
>>>>>>One idea for this work is to eventually provide access to alternate
>>>>>>datasets, for instance, a dataset that contains all the 3hr forecast
>>>>>>times from the different runs, or one that contained all the 12Z valid
>>>>>>times from the different runs. Tracking these detailed inventories is
>>>>>>just the first step but aggregation and alternate groupings of the
>>>>>>data
>>>>>>is pretty interesting to think about.
>>>>>>
>>>>>>How are your GRIB files structured and what kind of aggregation where
>>>>>>you thinking about?
>>>>>>
>>>>>>Ethan
>>>>>>
>>>>>>dan.swank wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>>Hello,
>>>>>>>
>>>>>>>I've been tinkering with the TDS aggregation capabilities and they
>>>>>>>work
>>>>>>>quite well for NetCDF data, however, I can't seem to find anything in
>>>>>>>the docs regarding aggregating GRIB.
>>>>>>>We want to get The NARR dataset which we have here at NCDC-NOMADS on
>>>>>>>the
>>>>>>>TDS.  It consists of hundreds of thousands of 50 Mb + GRIB files in a
>>>>>>>YYYYMM/YYYYMMDD tree.
>>>>>>> Just scouting for a quick answer here:
>>>>>>>Is aggregating the NARR GRIB currently feasable with the current
>>>>>>>release
>>>>>>>of TDS?  If so, do any docs exist which could give me a starting
>>>>>>>point?
>>>>>>>Converting it to NetCDF will not be possible (volume).
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>--
>>>Dan Swank <dan.swank at noaa.gov>
>>>NOMADS Project:  Software & Data Management
>>>Contractor - STG, Incorporated
>>>Veach-Baley Federal Building
>>>151 Patton Avenue
>>>Asheville, NC 28801-5001
>>>Phone: 828-271-4007
>>>
>>>==============================================================================
>>>To unsubscribe thredds, visit:
>>>http://www.unidata.ucar.edu/mailing-list-delete-form.html
>>>==============================================================================
>>>
>>
>>
>>
> 

==============================================================================
To unsubscribe thredds, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html
==============================================================================

More information about the Thredds mailing list