Re: [thredds] NcML aggregation strangely pulling files from multiple datasets

Hi John and Roy,

Yes, I see this problem too with a joinExisting aggregation of mooring data
where the last file in the aggregation is clobbered and recreated every two
hours.

To fix it I follow Roy's step 1 AND I reload the app with Tomcat's manager.
(I've found that I need to do both of these steps to correct the problem.)

If I've reloaded the app more than a few times I see the permgen error and
will need to restart Tomcat - most of the time I just restart tomcat after
removing the cache/agg files.

I¹ve struggled to consider a more permanent fix to this problem.  If anyone
has some suggestions my ears are wide open.

-Mike

-- 
Mike McCann 
Software Engineer 
Monterey Bay Aquarium Research Institute
7700 Sandholdt Road
Moss Landing, CA 95039-9644
Voice: 831.775.1769  Fax: 831.775.1736 http://www.mbari.org
 


On 5/19/11 3:59 PM, "John Maurer" <jmaurer@xxxxxxxxxx> wrote:

> MANY THANKS, ROY! That cleared up all the problems. I'm hugely relieved that I
> and/or my disk weren't going haywire. :) OK, catching my breath... phew.
> Cheers,
> John Maurer
> 
> On Thu, May 19, 2011 at 12:48 PM, Roy Mendelssohn <Roy.Mendelssohn@xxxxxxxx>
> wrote:
>> Hi John:
>> 
>> We have run into this too, particularly when adding new datasets.  Whenever
>> we do, we always do the following and find it clears up the problem:
>> 
>> 1.  Delete all of $TOMCAT_HOME/content/thredds/cache/agg
>> 
>> 2. Delete all of $TOMCAT_HOME/content/thredds/cache/ehcache
>> 
>> not certain ehculprit - it looks like the ehcache might assosciate the agg
>> cache with certain datasets, and if that gets changed the list gets out of
>> whack, but that is a wild guess.  What I do know is that when we have that
>> problem, those two steps clear it up.
>> 
>> The downside is that access to large aggregate datasets is slow the first
>> time they are accessed.
>> 
>> -Roy
>> 
>> On May 19, 2011, at 2:24 PM, John Maurer wrote:
>> 
>>> > Hi all,
>>> > I'm seeing some weird behavior today with one of our joinExisting NcML
>>> aggregations for high-frequency radio (HFR). The OPeNDAP page lists
>>> variables that are from a totally different dataset (e.g. HFR does not
>>> measure water temperature):
>>> http://oos.soest.hawaii.edu/thredds/idd/hfr.html?dataset=kokagg. I looked at
>>> "content/thredds/cache/agg/" at the aggregation cache file for the dataset
>>> (hioos-hfr-kokagg) and it confirms that it's strangely pulling files from 3
>>> unrelated datasets: a glider, an NCOM global FMRC, and the HFR data. Since
>>> nothing has changed recently in my config file for the dataset and I haven't
>>> recently upgraded TDS, this has me worried about a possible disk problem
>>> from recent power outages? The three datasets are not subdirectories of each
>>> other, so I don't see how TDS can be lumping them together. Any ideas what
>>> could be causing this?
>>> >
>>> > In addition to the above, 5 of my FMRC's are suddenly inaccessible today
>>> as well and were recently working just fine. It would seem their file
>>> aggregations may have similarly gone haywire as well, but their cache files
>>> under "content/thredds/collection/" (right?) are binary (.jdb) so I can't
>>> confirm whether they're combining multiple datasets. The catalogs give an
>>> error in threddsServlet.log with the message:
>>> >         at 
>>> ucar.nc2.ft.fmrc.FmrcDataset.getNetcdfDataset2D(FmrcDataset.java:166)
>>> >         at ucar.nc2.ft.fmrc.Fmrc.getDataset2D(Fmrc.java:183)
>>> >
>>> > What's the best way for me to clear all TDS caches? Is it safe to remove
>>> "content/thredds/cache/" entirely?, and then restart Tomcat/TDS? Will it get
>>> re-built then automatically after I access each dataset?
>>> >
>>> > I've copied the relevant agg and collection caches, config files, and
>>> threddsServlet.log to my anonymous ftp server at:
>>> >
>>> > ftp.soest.hawaii.edu <http://ftp.soest.hawaii.edu>
>>> > jmaurer/thredds/
>>> >
>>> > Thanks for your help!,
>>> > John Maurer
>>> > Pacific Islands Ocean Observing System (PacIOOS)
>>> > University of Hawaii
>>> > _______________________________________________
>>> > thredds mailing list
>>> > thredds@xxxxxxxxxxxxxxxx
>>> > For list information or to unsubscribe,  visit:
>>> http://www.unidata.ucar.edu/mailing_lists/
>> 
>> **********************
>> "The contents of this message do not reflect any position of the U.S.
>> Government or NOAA."
>> **********************
>> Roy Mendelssohn
>> Supervisory Operations Research Analyst
>> NOAA/NMFS
>> Environmental Research Division
>> Southwest Fisheries Science Center
>> 1352 Lighthouse Avenue
>> Pacific Grove, CA 93950-2097
>> 
>> e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address)
>> voice: (831)-648-9029
>> fax: (831)-648-8440
>> www: http://www.pfeg.noaa.gov/
>> 
>> "Old age and treachery will overcome youth and skill."
>> "From those who have been given much, much will be expected"
>> 
> 
> 
> 
> _______________________________________________
> thredds mailing list
> thredds@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe,  visit:
> http://www.unidata.ucar.edu/mailing_lists/