Nomads Forecast Aggregation

[John Caron]

Hi Steven:

try adding inherited="true":

<datasetFmrc name="gfs-test-FMRC" collectionType="ForecastModelRuns" harvest="true" path="fmrc/gfs_3">

        <metadata inherited="true">
                <dataFormat>GRIB-1</dataFormat>
                <serviceName>fmrcServices</serviceName>
        </metadata>


Steven.Anthony wrote:
> John and Ethan,
> The Nomads team here at NCDC has been looking into the FMRC capability
> of the TDS for a while.  I have tried many times with some success, but
> our main datasets still refuse to cooperate.  I feel that I am doing
> something wrong, but can't seem to fix it.  Can you take a look at our
> catalogs/data to see if there is something that I am doing wrong?
> 
> The Data:
> For my test, I have decided to use our GFS data, and then work from
> there if it works.
> The data is standard GFS one degree data from NCEP.  It is in GRIB-1
> format.  Each model run contains many files (one forecast hour per file)
> and we store the 4 runs from each day in a common folder.
> The DatasetScan has worked fantastically on this dataset, and is already
> in use.
> 
> The data is named by our own Nomads conventions:
> gfs_3_yyyymmdd_hhhh_fff.grb
> where:
> gfs_3 = model name & number
> yyyy = year
> mm = month
> dd = day
> hhhh = model run time (0000, 0600, 1200, 1800)
> fff = forecast hour
> 
> You can view the data and its structure at our public Nomads TDS. 
> http://nomads-d.ncdc.noaa.gov:8081/thredds/catalog.html
> in the GFS directory.
> 
> 
> The Catalog:
> First, I just updated our development server to the latest verstion
> (3.16.10 Build Date = 2007-06-14 00:14:30)
> Our DatasetScan catalogs still work, but our FMRC catalogs still only
> half work.
> 
> here is the truncated version of our catalog.xml file (working
> DatasetScans, comments, etc. removed):
> 
> <?xml version="1.0" encoding="UTF-8"?>
> <catalog
>     xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"
>     xmlns:xlink="http://www.w3.org/1999/xlink"
>     name="NCDC-NOMADS THREDDS Data Server II"
>     version="1.0.2">
> 
> <!-- Services -->
> <service name="fmrcServices" serviceType="Compound" base="">
>   <service name="ncdods" serviceType="OpenDAP" base="/thredds/dodsC/"/>
>   <service name="wcs" serviceType="WCS" base="/thredds/wcs/"/>
>   <service name="ncServer" serviceType="NetcdfServer"
> base="/thredds/ncServer/"/>
> </service>
> 
> 
> <datasetFmrc name="gfs-test-FMRC" collectionType="ForecastModelRuns"
> harvest="true" path="fmrc/gfs_3">
>         <metadata>
>                 <dataFormat>GRIB-1</dataFormat>
>                 <serviceName>fmrcServices</serviceName>
>         </metadata>
> 
> 
> 
>         <netcdf
> xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2"
> enhance="true">
>                 <aggregation dimName="run"
> type="forecastModelRunSingleCollection"
> fmrcDefinition="NCEP-GFS-Global_onedeg.fmrcDefinition.xml"
> timeUnitsChange="true" >
> 
>                         <scanFmrc
> location="/nomads3_data/raid2/noaaport/merged/gfs-hi/200705/20070531"
>                         regExp=".*\.grb$"
>                         runDateMatcher="#gfs_3_#yyyyMMdd_HH"
>                         forecastOffsetMatcher="HHH#.grb#"/>
> 
>                 </aggregation>
>         </netcdf>
> 
>         <fmrcInventory
> location="/nomads3_data/raid2/noaaport/merged/gfs-hi/200705/20070531"
> suffix=".grb" fmrcDefinition="NCEP-GFS-Global_onedeg.fmrcDefinition.xml" />
> </datasetFmrc>
> 
> </catalog>
> 
> 
> 
> This catalog works as in I don't receive an error in the web browser,
> however the models.log file has this this exception.
> [14/Jun/2007:10:58:11] ERROR thredds.catalog.InvDatasetFmrc : Error
> making catalog for fmrc/gfs_3
> java.lang.IllegalArgumentException: The service must not be null.
>         at
> thredds.cataloggen.CollectionLevelScanner.<init>(CollectionLevelScanner.java:237)
>         at
> thredds.cataloggen.StandardCatalogBuilder.setupAndScan(StandardCatalogBuilder.java:212)
>         at
> thredds.cataloggen.StandardCatalogBuilder.generateCatalog(StandardCatalogBuilder.java:133)
>         at
> thredds.cataloggen.DatasetScanCatalogBuilder.generateCatalog(DatasetScanCatalogBuilder.java:119)
>         at
> thredds.catalog.InvDatasetScan.makeCatalogForDirectory(InvDatasetScan.java:504)
>         at
> thredds.catalog.InvDatasetFmrc.makeCatalogScan(InvDatasetFmrc.java:445)
>         at
> thredds.catalog.InvDatasetFmrc.makeCatalog(InvDatasetFmrc.java:158)
>         at
> thredds.servlet.DataRootHandler.makeDynamicCatalog(DataRootHandler.java:1167)
>         at
> thredds.servlet.DataRootHandler.getCatalog(DataRootHandler.java:1135)
>         at
> thredds.servlet.DataRootHandler.processReqForCatalog(DataRootHandler.java:1050)
>         at thredds.servlet.CatalogServlet.doGet(CatalogServlet.java:78)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
>         at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
>         at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
>         at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
>         at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
>         at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:432)
>         at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
>         at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
>         at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
>         at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
>         at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
>         at
> org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
>         at
> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
>         at
> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
>         at
> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
>         at java.lang.Thread.run(Thread.java:595)
> 
> This sounds about right as the FMRC is parsing the dates right, and
> generating the catalog, however, it does not provide the access links
> that should be there (OpenDAP, WCS and NetCDF Server)
> 
> If you want any more information, feel free to ask.
> I appreciate your level of support and commitment, and we look forward
> to getting the FMRC aggregation working for all of our datasets so we
> can start to use it more.
> 
> Thanks again
>  - Steven
> 
> 
>