[thredds] FMRC

I have to work with a lot of data, which are structured differently (in
naming and location conventions)
from a product to another one. I wonder if all of these would be easilly
configurable or if we have to
change their organisation. So I have a few questions:

- I would like to know more about all the scan fonctionnalities:
    Are they working the same way for matching regExp, and is the full file
path tested by this regExp or only the file name ?
    If the entire path is tested, does dateFormatMark, runDateMatcher and
other date markers use the full path to ?
    In other word, is it possible to extract date informations from
directory name as from file name ?

- In which cases netcdf files are opened when building catalogs, and if I
provide needed metadata,
    will they be opened to (optimizations)?

- I have a problem with the following datasetFmrc:

    Dataset definition:
    -------------------

    <datasetFmrc name="TestDatasetFmrc"  path="testDatasetFmrc">
        <metadata inherited="true">
            <serviceName>multiService</serviceName>
            <dataType>Grid</dataType>
            <dataFormat>NetCDF</dataFormat>
        </metadata>
        <netcdf xmlns="
http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2"; enhance="true">
            <aggregation dimName="run" type="forecastModelRunCollection"
recheckEvery="1 hour">
                <scan
location="/home/coriolis_exp/spool/co01/co0123/co012302/co01230204/archive/"
                            regExp="(.*)MM5_200710([0-9]{2})\.nc$"
                            suffix=".nc" subdirs="true"
                            dateFormatMark="MM5_#yyyyMMdd"
                            olderThan="5 min" />
            </aggregation>
        </netcdf>
    </datasetFmrc>

    The files are all in the same directory:
    ----------------------------------------

    |-- /home/coriolis_exp/spool/co01/co0123/co012302/co01230204/archive/
        |-- ...
        |-- MM5_20071008.nc
        |-- MM5_20071009.nc
        |-- ...

    These files are structured as below (ncdump -h):
    ------------------------------------------------

    netcdf MM5_20071009 {
    dimensions:
                    lon = 440 ;
                    lat = 185 ;
                    time = UNLIMITED ; // (41 currently)
    variables:
                    float lon(lon) ;
                                    lon:long_name = "longitude" ;
                                    lon:units = "degree_east" ;
                    float lat(lat) ;
                                    lat:long_name = "latitude" ;
                                    lat:units = "degree_north" ;
                    double time(time) ;
                                    time:long_name = "seconds since the
origin date" ;
                                    time:time_origin = "01-JAN-0000
00:00:00" ;
                                    time:units = "seconds" ;
                    float uwd(time, lat, lon) ;
                                    uwd:long_name = "10-meter U Component" ;
                                    uwd:units = "m/s" ;
                    float vwd(time, lat, lon) ;
                                    vwd:long_name = "10-meter V Component" ;
                                    vwd:units = "m/s" ;
                    float pmer(time, lat, lon) ;
                                    pmer:long_name = "Sea level pressure" ;
                                    pmer:units = "Pa" ;
                    float rad(time, lat, lon) ;
                                    rad:long_name = "solar radiation flux" ;
                                    rad:units = "w/m^2" ;
                    float sat(time, lat, lon) ;
                                    sat:long_name = "2-meter Temperature" ;
                                    sat:units = "C degrees" ;
                    float rain(time, lat, lon) ;
                                    rain:long_name = "precipitation during
one period" ;
                                    rain:units = "kg/m^2" ;
                    float hflw(time, lat, lon) ;
                                    hflw:long_name = "long wave heat flux" ;
                                    hflw:units = "w/m^2" ;
                    float rh(time, lat, lon) ;
                                    rh:long_name = "Relative humidity" ;
                                    rh:units = "%" ;

    // global attributes:
                    :Conventions = "COARDS" ;
                    :host = "ftp.unidata.ucar.edu" ;
                    :directory = "pub/netcdf/Conventions/COARDS" ;
                    :title = "MM5 version 3 format output on sigma levels" ;
                    :history = "Tue Oct  9 04:46:11 2007: ncrename -d
j_cross,lon final.nc\n",
                                    "Tue Oct  9 04:46:11 2007: ncrename -d
i_cross,lat final.nc\n",
                                    "Tue Oct  9 04:46:11 2007: ncatted -O -a
history,global,o,c, final.nc\n",
                                    "" ;
                    :institution = "CETEMPS - University of L\'Aquila,
Italy" ;
                    :source = "NCAR MM5 model version 3" ;
                    :references = "http://www.mmm.ucar.edu/mm5/mm5-home.html";
;
                    :comment = "No comments yet" ;

    I can access the dataset with the web browser:
    ----------------------------------------------

    TestDatasetFmrc
        Forecast Model Run Collection (2D time coordinates)
        Best Time Series
        Forecast Model Run
        Constant Forecast Offset
        Constant Forecast Date

    But when I try to get any of the proposed datasets, it results in:
    ------------------------------------------------------------------

    "Etat HTTP 400 - NULL message java.lang.NullPointerException" in the
browser

    And in the threddsServlet.log:
    ------------------------------

    2007-10-09T14:31:42.373 +0200 [   3177637][      45] ERROR -
thredds.servlet.ServletUtil - handleException
    java.lang.NullPointerException
    at thredds.servlet.ViewServlet$IDV.getViewerLinkHtml(ViewServlet.java
:212)
    at thredds.servlet.ViewServlet.showViewers(ViewServlet.java:159)
    at thredds.servlet.CatalogServicesServlet.showDataset(
CatalogServicesServlet.java:324)
    at thredds.servlet.CatalogServicesServlet.handleCatalogServiceRequest(
CatalogServicesServlet.java:201)
    at thredds.servlet.CatalogServicesServlet.doGet(
CatalogServicesServlet.java:98)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
ApplicationFilterChain.java:252)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(
ApplicationFilterChain.java:173)
    at org.apache.catalina.core.StandardWrapperValve.invoke(
StandardWrapperValve.java:213)
    at org.apache.catalina.core.StandardContextValve.invoke(
StandardContextValve.java:178)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(
AuthenticatorBase.java:432)
    at org.apache.catalina.core.StandardHostValve.invoke(
StandardHostValve.java:126)
    at org.apache.catalina.valves.ErrorReportValve.invoke(
ErrorReportValve.java:105)
    at org.apache.catalina.core.StandardEngineValve.invoke(
StandardEngineValve.java:107)
    at org.apache.catalina.connector.CoyoteAdapter.service(
CoyoteAdapter.java:148)
    at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java
:869)
    at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection
(Http11BaseProtocol.java:664)
    at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(
PoolTcpEndpoint.java:527)
    at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(
LeaderFollowerWorkerThread.java:80)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(
ThreadPool.java:684)
    at java.lang.Thread.run(Thread.java:595)


    I can't see what's going wrong...

  Thanks for your help.
  Jonathan
  • 2007 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: