[thredds] TDS not refreshing catalog after TDM trigger

• To: thredds@xxxxxxxxxxxxxxxx
• Subject: [thredds] TDS not refreshing catalog after TDM trigger
• From: Arnaud Dumont <dumont@xxxxxxxx>
• Date: Fri, 20 Mar 2020 08:23:02 -0600

I'm struggling to get TDS to update its indices for a realtime GRIB
dataset. I'm running


  version= 4.6.14 - 2019-07-23T11:04:31-0600

  tdmFat-4.6.14.jar


with the catalog.xml featureCollection set up as:

  <featureCollection name="gtgn" featureType="GRIB2" path="gtgn">


    <metadata inherited="true">

      <serviceName>all</serviceName>

      <dataType>Grid</dataType>

    </metadata>



    <collection name="gtgn"

spec="/d1/data_archive/gtg_ncst_grib/**/gtgn.*\.grb2$"

timePartition="file"

                dateFormatMark="gtg_ncst_grib/#yyyyMMdd'/gtgn.t'HHmm#" />



    <update startup="never" trigger="allow"/>

    <tdm rewrite="test" rescan="0 0/5 * * * ? *" trigger="allow" />


  </featureCollection>

When I run the tdm with the following args:

  java -Xmx4g -Dtds.content.root.path=/d1/data_archive -jar
/d1/data_archive/thredds/tdmFat-4.6.14.jar  -cred "user:password" -log
DEBUG &> /d1/data_archive/tdm/runTdm.log

I see that it correctly regenerates the indicies and triggers:


2020-03-20T13:19:48.644 +0000 DEBUG -  createIndex for
/d1/data_archive/gtg_ncst_grib/20200320/gtgn.t0730z.edr.f000.grb2.ncx3

2020-03-20T13:19:48.644 +0000 DEBUG -   write RecordMaps: bytes = 675
record = 52 bytesPerRecord=12

2020-03-20T13:19:48.645 +0000 DEBUG -   write GribCollectionIndex= 700 bytes

2020-03-20T13:19:48.645 +0000 DEBUG - That took 152 msecs

2020-03-20T13:19:48.658 +0000 DEBUG -      Using canonical partition
gtgn.t1300z.edr.f000.grb2

2020-03-20T13:19:48.681 +0000 INFO  - RewriteFilePartition gtgn-20200320
took 6648 msecs

2020-03-20T13:19:48.898 +0000 DEBUG -      Using canonical partition
gtgn-20200319

2020-03-20T13:19:48.988 +0000 INFO  - updateGribCollection gtgn changed
true took 20896 msecs

2020-03-20T13:19:48.989 +0000 DEBUG - 2020-03-20T13:19:48.988Z gtgn changed
true

2020-03-20T13:19:49.013 +0000 INFO  - send trigger to
http://localhost:8080/thredds/admin/collection/trigger?trigger=never&collection=gtgn
status = 200

2020-03-20T13:20:01.609 +0000 INFO  - updateGribCollection gtgn changed
false took 1269 msecs

2020-03-20T13:20:01.610 +0000 DEBUG - 2020-03-20T13:20:01.609Z gtgn changed
false


However, the catalog (below) never updates:


  http://poseidon.rap.ucar.edu:8080/gtgnTDS/catalog/gtgn/catalog.html


even though the WMS getCapabilities (below) DOES update for the new files:



http://poseidon.rap.ucar.edu:8080/thredds/wms/gtgn/TP?service=WMS&version=1.3.0&request=GetCapabilities


I'm relying on the xml catalog to determine run times for some other
forecast datasets, so having updated information is critical to my display
application.

...and a seemingly unrelated oddity is that the getCapabilities' times are
all rounded to the hour, even though the files are 15-minutely. The HTML
catalog gets the times correct (when it's updating correctly, that is), but
the getCapabilities has each 15-minute file listed as being at the top of
the hour (dropping minutes field?). See the latest values in the "time"
dimension below:

<Layer queryable="1">
<Name>Eddy_dissipation_parameter_altitude_above_msl</Name>
<Title>
Eddy dissipation parameter @ Specific altitude above mean sea level
</Title>
<Abstract>
Eddy dissipation parameter @ Specific altitude above mean sea level
</Abstract>
<EX_GeographicBoundingBox>
<westBoundLongitude>-139.97003181300516</westBoundLongitude>
<eastBoundLongitude>-57.26792343693751</eastBoundLongitude>
<southBoundLatitude>16.20865459974452</southBoundLatitude>
<northBoundLatitude>55.51688759604331</northBoundLatitude>
</EX_GeographicBoundingBox>
<BoundingBox CRS="CRS:84" minx="-139.97003181300516" maxx="
-57.26792343693751" miny="16.20865459974452" maxy="55.51688759604331"/>
<Dimension name="elevation" units="m" default="0.0">
0.0,30.0,304.0,609.0,914.0,1219.0,1524.0,1828.0,2133.0,2438.0,2743.0,3048.0,3352.0,3657.0,3962.0,4267.0,4572.0,4876.0,5181.0,5486.0,5791.0,6096.0,6400.0,6705.0,7010.0,7315.0,7620.0,7924.0,8229.0,8534.0,8839.0,9144.0,9448.0,9753.0,10058.0,10363.0,10668.0,10972.0,11277.0,11582.0,11887.0,12192.0,12496.0,12801.0,13106.0,13411.0,13716.0,14020.0,14325.0,14630.0,14935.0,15240.0
</Dimension>
<Dimension name="time" units="ISO8601" multipleValues="true" current="true"
default="2020-03-20T13:00:00.000Z">
2019-11-13T19:00:00.000Z,2019-11-13T20:00:00.000Z,2019-11-13T21:00:00.000Z,2019-11-13T22:00:00.000Z,
...
2020-03-20T11:00:00.000Z,2020-03-20T11:00:00.000Z,2020-03-20T11:00:00.000Z,2020-03-20T11:00:00.000Z,2020-03-20T12:00:00.000Z,2020-03-20T12:00:00.000Z,2020-03-20T12:00:00.000Z,2020-03-20T12:00:00.000Z,2020-03-20T13:00:00.000Z,2020-03-20T13:00:00.000Z,2020-03-20T13:00:00.000Z,2020-03-20T13:00:00.000Z

</Dimension>


Any ideas on what I might be doing wrong or suggestions on how to work
around these issues?

Thanks,
Arnaud
---
Arnaud Dumont
Research Applications Laboratory (RAL)
National Center for Atmospheric Research (NCAR)
3450 Mitchell Ln, Boulder CO, 80301
dumont@xxxxxxxx
+1(303)497-8434

• Follow-Ups:
  • Re: [thredds] TDS not refreshing catalog after TDM trigger
    • From: Sean Arms