Re: [thredds] Possible caching problems?

Hi Pauline,

Rich wrote:
> This was introduced in mid-September, so you may need to upgrade your
> TDS.

The fix Rich mentions is only in our 4.1 (development) release, not in
our 4.0 (stable) release. The caching system has undergone some big
changes in 4.1 so I'm not sure if this fix will ever make it into 4.0.

Can you upgrade this system to TDS 4.1?

Ethan

Richard Signell wrote:
> Pauline,
> 
> To keep files that are overwritten with new data from being cached in
> the TDS, you can add a tag cache="false" in the datasetRoot element.
> This was introduced in mid-September, so you may need to upgrade your
> TDS.  Here's an example catalog which serves two ocean model forecast
> files.  These forecast files are replaced every day with new files,
> but they always have the same name.
> 
> 
> <?xml version="1.0" encoding="UTF-8"?>
> <catalog 
> xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0";
>   xmlns:xlink="http://www.w3.org/1999/xlink"; name="THREDDS Catalog for
> NetCDF Files" version="1.0.1">
> 
>   <service name="allServices" serviceType="Compound" base="">
>     <service name="ncdods" serviceType="OpenDAP" base="/thredds/dodsC/"/>
>     <service name="HTTPServer" serviceType="HTTPServer"
> base="/thredds/fileServer/"/>
>   </service>
> 
>   <datasetRoot path="FVCOM" location="/http/www/CODFISH/Data/FVCOM/"
> cache="false"/>
> 
>   <dataset name="GOM2 Forecast with no caching" ID="gom2_nocache"
> serviceName="allServices"
>     urlPath="FVCOM/NECOFS/Forecasts/NECOFS_GOM2_FORECAST.nc" dataType="Grid"/>
>   <dataset name="MASSBAY Forecast with no caching"
> ID="massbay_nocache" serviceName="allServices"
>     urlPath="FVCOM/NECOFS/Forecasts/NECOFS_FVCOM_OCEAN_MASSBAY_FORECAST.nc"
> dataType="Grid"/>
> 
> </catalog>
> 
> -Rich
> 
> On Wed, Dec 2, 2009 at 12:20 AM, Pauline Mak <pauline.mak@xxxxxxxxxxx> wrote:
>> Hi all,
>>
>> I have a user of TDS who had updated some files on the server and it seems
>> like TDS is caching this and not refreshing it.  Has anyone else come across
>> this problem?
>>
>> The directory we're looking at is:
>>
>> http://opendap-ivec.arcs.org.au/thredds/catalog/IMOS/SRS/SeaDAS/test/
>>
>> As a test we copied one of the problematic files
>> (A2009100.0320.L2_OCEANGRID.hdf) and called it test.hdf, and it works fine!
>>  Note that, while the files are identical, the metadata in the global
>> attributes is different:
>>
>> new file: time: "2009-04-10T03:22:38Z"
>> old file: time: "2009-03-10T03:22:38Z"
>>
>> Interestingly, for A2009100.0320.L2_OCEANGRID.hdf, it seems like there's
>> problem with the dimensions (which would explain why WCS and WMS doesn't
>> work).  When I tried to retrieve the first 2000 elements in latitude, it
>> worked fine (I got the ascii response).  However, getting 2200, out of a
>> possible 2437, I get the stacktrace error attached.
>>
>> I am using version 4.0.26 - 20090831.2140 running on Jetty 6.1.18, with the
>> following cache settings in TDS:
>>
>>  <DiskCache>
>>    <alwaysUse>true</alwaysUse>
>>    <dir>/data/tmp/thredds/cache/</dir>
>>    <scour>1 hour</scour>
>>    <maxSize>10 Gb</maxSize>
>>  </DiskCache>
>>
>>  <NetcdfFileCache>
>>    <minFiles>200</minFiles>
>>    <maxFiles>400</maxFiles>
>>    <scour>10 min</scour>
>>  </NetcdfFileCache>
>>
>>  <NetcdfDatasetCache>
>>    <minFiles>100</minFiles>
>>    <maxFiles>200</maxFiles>
>>    <scour>10 min</scour>
>>  </NetcdfDatasetCache>
>>
>>  <NetcdfSubsetService>
>>    <allow>true</allow>
>>    <dir>/data/tmp/thredds/ncSubsetCache/</dir>
>>    <scour>10 min</scour>
>>    <maxAge>-1 min</maxAge>
>>    <metarDataDir>/opt/tomcat/content/thredds/public/stn/</metarDataDir>
>>    <metarRawDir>/data/ldm/pub/decoded/netcdf/surface/metar/</metarRawDir>
>>  </NetcdfSubsetService>
>>
>>  <WCS>
>>    <allow>true</allow>
>>    <allowRemote>false</allowRemote>
>>    <dir>/tmp/thredds/wcsCache/</dir>
>>    <scour>15 min</scour>
>>    <maxAge>30 min</maxAge>
>>  </WCS>
>>
>>  <WMS>
>>      <allow>true</allow>
>>
>> <paletteLocationDir>/opt/tds/webapps/thredds/WEB-INF/palettes</paletteLocationDir>
>>  </WMS>
>>
>> Thanks!
>>
>> -Pauline.
>>
>> --
>> Pauline Mak
>>
>> Assistant Manager, ARCS Data Services
>> Ph:  +61 3 6226 7518
>> Mob: +61 411 638 196
>> Email: pauline.mak@xxxxxxxxxxx
>> Jabber: pauline.mak@xxxxxxxxxxx
>> Calendar: http://tinyurl.com/pmak-arcs-calendar
>> http://www.arcs.org.au/
>>
>> TPAC
>> Email: pauline.mak@xxxxxxxxxxx
>> http://www.tpac.org.au/
>>
>>
>>
>>
>> java.lang.RuntimeException: NcSDArray
>> ArrayIndexOutOfBoundsException=NcSDArray read Latitude Latitude(0,1,2200)
>> null dataset= IMOS/SRS/SeaDAS/test/A2009100.0320.L2_OCEANGRID.hdf
>>        at thredds.server.opendap.NcSDArray.read(NcSDArray.java:118)
>>        at opendap.servlet.AsciiWriter.writeAsc(AsciiWriter.java:95)
>>        at opendap.servlet.AsciiWriter.toASCII(AsciiWriter.java:56)
>>        at
>> thredds.server.opendap.OpendapServlet.doGetASC(OpendapServlet.java:320)
>>        at
>> thredds.server.opendap.OpendapServlet.doGet(OpendapServlet.java:222)
>>        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>        at
>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
>>        at
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
>>        at thredds.servlet.filter.CookieFilter.doFilter(CookieFilter.java:54)
>>        at
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1148)
>>        at
>> thredds.servlet.filter.RequestQueryFilter.doFilter(RequestQueryFilter.java:121)
>>        at
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1148)
>>        at
>> thredds.servlet.filter.RequestPathFilter.doFilter(RequestPathFilter.java:105)
>>        at
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1148)
>>        at
>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:387)
>>        at
>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>        at
>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>>        at
>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
>>        at
>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
>>        at
>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>        at
>> org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
>>        at
>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>        at org.mortbay.jetty.Server.handle(Server.java:326)
>>        at
>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
>>        at
>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
>>        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:539)
>>        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>        at
>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
>>        at
>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:520)
>>
>>
>> _______________________________________________
>> thredds mailing list
>> thredds@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe,  visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>>
> 
> 
> 



  • 2009 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: