Re: [thredds] aggregation caching: cache cleared when Tomcat restarted?

Hi John,
I'm afraid that none of the settings to AggregationCache, or removing it 
entirely fix, have solved the recurring slowness of my joinExisting 
aggregations. I have separately tried the following without success:
- removing AggregationCache settings from threddsServlet.log
- setting scour to -1
- setting maxAge to a very long period like "1 year"

In all cases, I restarted Tomcat and revisited those joinExisting aggregations 
to initialize the cache. Upon returning at a later time (1 day or sometimes a 
few days) or sometimes after another Tomcat restart, I would eventually find 
that access to these aggregations was again slow. When I then look at cache/agg 
as you suggested, the aggregation files for these particular datasets have been 
updated after the slowness. However, their contents were no different than 
before they were reconstructed, indicating that nothing had changed with the 
dataset--which is the case: no files have been modified, added, or removed from 
the collection for over a month now. Maybe a bug?, or perhaps something else I 
haven't tried or have misunderstood?
Thanks!,
John Maurer

----- Original Message -----
From: "John Maurer, IV" <jmaurer@xxxxxxxxxx>
Date: Tuesday, December 21, 2010 1:54 pm
Subject: Re: [thredds] aggregation caching: cache cleared when Tomcat restarted?
To: John Caron <caron@xxxxxxxxxxxxxxxx>
Cc: Unidata netCDF Java Support <support-netcdf-java@xxxxxxxxxxxxxxxx>

> OK, thanks John. I have just removed the AggregationCache settings from my 
> threddsConfig.xml, restarted Tomcat, and accessed these particular 
> aggregations. I will keep an eye on their performance in the coming week(s) 
> in the hopes they will remain swift.
> Cheers,
> John Maurer
> 
> ----- Original Message -----
> From: John Caron <caron@xxxxxxxxxxxxxxxx>
> Date: Tuesday, December 21, 2010 9:56 am
> Subject: Re: [thredds] aggregation caching: cache cleared when Tomcat 
> restarted?
> To: "John Maurer, IV" <jmaurer@xxxxxxxxxx>
> Cc: Unidata netCDF Java Support <support-netcdf-java@xxxxxxxxxxxxxxxx>
> 
> 
> 
| 

> > If you have some aggs that never change, I would remove
> > 
> >   <AggregationCache>
> >     <scour>24 hours</scour>
> >     <maxAge>30 days</maxAge>
> >   </AggregationCache>
> > 
> > all together. Otherwise, make maxAge longer than the longest time between 
> > changes, eg maybe one year or something. 
> > 
> > basically, you dont want to remove active aggregations.
> > 
> > Restart tomcat, and make sure all aggregation get hit. After that, watch 
> > closely for the slowdown and keep an eye on cache/agg.
> > 
> > On 12/21/2010 11:45 AM, John Maurer, IV wrote: 
Hi John,
> > The last time a file was modified in that dataset's data directory is 11/28 
> > (full list below). I upgraded to TDS 4.2 on 11/29. My threddsConfig.xml is 
> > attached.
> > Thanks!,
> > John Maurer
> > 
> > [root@lawelawe:/export/lawelawe1/model/tide/netcdf_data/mhi/elev]# ls -latr
> > total 17224552
> > -rw-r--r-- 1 root root 375230252 Nov  1 14:43 MHIelev_2008_01.nc
> > -rw-r--r-- 1 root root 351022124 Nov  1 14:43 MHIelev_2008_02.nc
> > -rw-r--r-- 1 root root 375230252 Nov  1 14:43 MHIelev_2008_03.nc
> > -rw-r--r-- 1 root root 363126188 Nov  1 14:43 MHIelev_2008_04.nc
> > -rw-r--r-- 1 root root 375230252 Nov  1 14:43 MHIelev_2008_05.nc
> > -rw-r--r-- 1 root root 363126188 Nov  1 14:44 MHIelev_2008_06.nc
> > -rw-r--r-- 1 root root 375230252 Nov  1 14:44 MHIelev_2008_07.nc
> > -rw-r--r-- 1 root root 375230252 Nov  1 14:44 MHIelev_2008_08.nc
> > -rw-r--r-- 1 root root 363126188 Nov  1 14:44 MHIelev_2008_09.nc
> > -rw-r--r-- 1 root root 375230252 Nov  1 14:44 MHIelev_2008_10.nc
> > -rw-r--r-- 1 root root 363126188 Nov  1 14:44 MHIelev_2008_11.nc
> > -rw-r--r-- 1 root root 375230252 Nov  1 14:44 MHIelev_2009_01.nc
> > -rw-r--r-- 1 root root 338918060 Nov  1 14:44 MHIelev_2009_02.nc
> > -rw-r--r-- 1 root root 375230252 Nov  1 14:44 MHIelev_2009_03.nc
> > -rw-r--r-- 1 root root 363126188 Nov  1 14:44 MHIelev_2009_04.nc
> > -rw-r--r-- 1 root root 375230252 Nov  1 14:45 MHIelev_2009_05.nc
> > -rw-r--r-- 1 root root 363126188 Nov  1 14:45 MHIelev_2009_06.nc
> > -rw-r--r-- 1 root root 375230252 Nov  1 14:45 MHIelev_2009_07.nc
> > -rw-r--r-- 1 root root 375230252 Nov  1 14:45 MHIelev_2009_08.nc
> > -rw-r--r-- 1 root root 363126188 Nov  1 14:45 MHIelev_2009_09.nc
> > -rw-r--r-- 1 root root 375230252 Nov  1 14:45 MHIelev_2009_10.nc
> > -rw-r--r-- 1 root root 363126188 Nov  1 14:45 MHIelev_2009_11.nc
> > -rw-r--r-- 1 root root 375230252 Nov  1 14:45 MHIelev_2010_01.nc
> > -rw-r--r-- 1 root root 338918060 Nov  1 14:46 MHIelev_2010_02.nc
> > -rw-r--r-- 1 root root 375230252 Nov  1 14:46 MHIelev_2010_03.nc
> > -rw-r--r-- 1 root root 363126188 Nov  1 14:46 MHIelev_2010_04.nc
> > -rw-r--r-- 1 root root 375230252 Nov  1 14:46 MHIelev_2010_05.nc
> > -rw-r--r-- 1 root root 363126188 Nov  1 14:46 MHIelev_2010_06.nc
> > -rw-r--r-- 1 root root 375230252 Nov  1 14:46 MHIelev_2010_07.nc
> > -rw-r--r-- 1 root root 375230252 Nov  1 14:46 MHIelev_2010_08.nc
> > -rw-r--r-- 1 root root 363126188 Nov  1 14:46 MHIelev_2010_09.nc
> > -rw-r--r-- 1 root root 375230252 Nov  1 14:47 MHIelev_2010_10.nc
> > -rw-r--r-- 1 root root 363126188 Nov  1 14:47 MHIelev_2010_11.nc
> > -rw-r--r-- 1 root root 375230252 Nov  1 14:47 MHIelev_2011_01.nc
> > -rw-r--r-- 1 root root 338918060 Nov  1 14:47 MHIelev_2011_02.nc
> > -rw-r--r-- 1 root root 375230252 Nov  1 14:47 MHIelev_2011_03.nc
> > -rw-r--r-- 1 root root 363126188 Nov  1 14:47 MHIelev_2011_04.nc
> > -rw-r--r-- 1 root root 375230252 Nov  1 14:47 MHIelev_2011_05.nc
> > -rw-r--r-- 1 root root 363126188 Nov  1 14:48 MHIelev_2011_06.nc
> > -rw-r--r-- 1 root root 375230252 Nov  1 14:48 MHIelev_2011_07.nc
> > -rw-r--r-- 1 root root 375230252 Nov  1 14:48 MHIelev_2011_08.nc
> > -rw-r--r-- 1 root root 363126188 Nov  1 14:48 MHIelev_2011_09.nc
> > -rw-r--r-- 1 root root 375230252 Nov  1 14:48 MHIelev_2011_10.nc
> > -rw-r--r-- 1 root root 363126188 Nov  1 14:48 MHIelev_2011_11.nc
> > drwxr-xr-x 4 root root      8192 Nov 15 13:36 ..
> > -rw-r--r-- 1 root root 363630524 Nov 26 15:10 MHIelev_2008_12.nc
> > -rw-r--r-- 1 root root 363630524 Nov 26 15:10 MHIelev_2009_12.nc
> > -rw-r--r-- 1 root root 363630524 Nov 26 15:33 MHIelev_2010_12.nc
> > -rw-r--r-- 1 root root      2216 Nov 26 15:45 velocity.log.1
> > -rw-r--r-- 1 root root     55699 Nov 26 16:16 velocity.log
> > -rw-r--r-- 1 root root 363630524 Nov 28 21:46 MHIelev_2011_12.nc
> > drwxr-xr-x 2 root root      4096 Nov 28 21:51 .
> > 
> > 
> > ----- Original Message -----
> > From: John Caron <caron@xxxxxxxxxxxxxxxx>
> > Date: Monday, December 20, 2010 3:33 pm
> > Subject: Re: [thredds] aggregation caching: cache cleared when Tomcat 
> > restarted?
> > To: "John Maurer, IV" <jmaurer@xxxxxxxxxx>
> > 
> 
> 
-----------------------------------------------------------



> 
> |-----------------------------------------------------------



-----------------------------------------------------------
| 
> > Hi John:
> > > 
> > > So if I look at the agg cache for 
> > > 
> > > http://oos.soest.hawaii.edu/thredds/idd/tide_mod.html?dataset=tide_elev_mhi
> > > 
> > > which is in content/thredds/cache/agg/hioos-tide_mhi-elev
> > > 
> > > the date on that appears to be 12/20 9:31 am.
> > > 
> > > so: 
> > > 
> > > 1) when was the last time any file in that collection changed, if you 
> > > look at lastModified date from ls -a ?
> > > 2) can i see your threddsConfig.xml ?
> > > 
> > > 
> > > On 12/16/2010 12:58 PM, John Maurer, IV wrote: 
Hi John,
> > > Some of my aggregations are updated with a new file daily but others are 
> > > more permanent. Because of the daily updates, I need to keep my scour set 
> > > to every 24 hours. However, the dataset that is more permanent (our tide 
> > > model, which has forecasts out through the end of 2011) is the one 
> > > causing me the recurring slowness. Even though the files in that dataset 
> > > have not been updated since late November nor have any new files been 
> > > created, it is still taking several minutes to access the OPeNDAP URL on 
> > > what seems to be a daily basis, though I'm still trying to tease out what 
> > > the pattern is. The file for that dataset gets re-written in the 
> > > content/thredds/cache/agg folder when this happens. It'll be fine for a 
> > > day, but then by the next day it's usually slow again on first access (~5 
> > > minutes to access the OPeNDAP page). It would appear that the scour is 
> > > happening regardless of my 30-day maxAge setting? Don't know what else I 
> > > can send you to help troubleshoot; lemme know... In case it helps, here 
> > > are the datasets that are causing me this problem:
> - > > > 
> http://oos.soest.hawaii.edu/thredds/idd/tide_mod.html?dataset=tide_elev_mhi
- > > > 
http://oos.soest.hawaii.edu/thredds/idd/tide_mod.html?dataset=tide_velo_mhi
- > > > 
http://oos.soest.hawaii.edu/thredds/idd/tide_mod.html?dataset=tide_elev_bi
- > > > 
http://oos.soest.hawaii.edu/thredds/idd/tide_mod.html?dataset=tide_velo_bi

Thanks,
> > > John Maurer 
> > > 
> > > > Hi John:
> > > > 
> > > > 1) Yes, this says every 24 hours, remove files with lastModified 
> > > > date > 
> > > > 30 days. If your aggregations are more permanent, change to a 
> > > > longer 
> > > > time, or set scour to -1 to not scour at all.
> > > > 
> > > > 2) Look under content/thredds/cache/agg:
> > > > 
> > > > each joinExisting should have an XML file in there. the content 
> > > > should 
> > > > be sort of obvious, send it to me if not. anyway, it will get 
> > > > updated 
> > > > when the file collection changes. otherwise, it should survive a 
> > > > restart 
> > > > and should make things fast if most of the files havent changed.
> > > > 
> > > > let me know if it seems something is not working as described.
> > >  > _______________________________________________ > thredds mailing list 
> > > > thredds@xxxxxxxxxxxxxxxx > For list information or to unsubscribe,  
> > > visit: http://www.unidata.ucar.edu/mailing_lists/ 

> | -----------------------------------------------------------





  • 2010 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: