Hi Tom,
Thanks for the further details.
We use the Spring Log4jWebConfigurer to configure log4j and call the
shutdownLogging() call when the TDS is shutdown. The shutdownLogging()
method calls LogManager.shutdown(). So, seems like either the log4j code
isn't shutting down properly or Tomcat is catching these threads before
they have time to complete their shutdown.
I've reopened this issue on our list but we won't be able to spend much
time on it for awhile. We do have plans to test the TDS on Tomcat 7,
we'll see if that helps.
Ethan
On 4/19/2011 3:42 AM, Tom Kunicki wrote:
>
> Hi Roy and Ethan,
>
> Just for clarity this is a more of a warning to indicated the
> presence of a PermGen leak. What a PermGen leak means in this context is that
> the
> classes associated with the TDS war can not be unloaded from PermGen
> memory. This is of no consequence unless the application is undeployed
> or redeployed during a given runtime of the JVM. Because the classes for
> the TDS war cannot be unloaded the classes loaded for each deployment
> stay in memory. After a certain number of deployment cycles PermGen
> memory will be exhausted and the JVM will effectively become
> unresponsive (classes can't be loaded, strings can't be interned, etc).
>
> Longer explanation of this specific error...
>
> The specific message states that a value stored in a ThreadLocal
> instance is a class loaded from the TDS war. The implementation of
> ThreadLocal (under the JRE hood) uses a static variable in a class
> loaded by the system class loader (present the entire JVM runtime). Now
> since this value is loaded by the TDS webapp class loader and there is a
> reference to this value held by a ThreadLocal (loaded by the system
> class loader) and every class instance has a reference to it's class
> loader (and the class loader has a reference to all the classes it's
> loaded) the classes loaded for the TDS deployment can now never be unloaded.
>
> The way to avoid this is to *always* call remove when ThreadLocal is
> used. Practically speaking many of these leaks are introduced by
> dependencies (like Log4J) but many of these jars offer clean-up
> routines. The error below is introduced by Log4J and one should be able
> to call org.apache.log4j.LogManager.shutdown() on servlet unloading to
> force it to release all it's resources.
>
> These are very difficult leaks to track down and they are often (but
> not always) introduced by dependencies. I would hope some effort would
> be used to track these down and remove them as often times restarting a
> JVM server on redeploy is not an ideal solution.
>
> Tom
>
>
>
> On Apr 18, 2011, at 6:19 PM, Ethan Davis wrote:
>
>> Hi Roy,
>>
>> Yes, the TDS gets this type of messages on TDS shutdown when running
>> under Tomcat 6.0.24 or later. Tomcat looks for memory (thread) leaks
>> when web applications are stopped and tries to shut the threads down.
>> Tomcat evidently started detecting this particular type of thread
>> leakage in Tomcat 6.0.24 but proper shutdown of them isn't available
>> until Tomcat 7.0.6 [1]. It does not appear that these threads are leaked
>> except upon shutdown.
>>
>> This could be an issue if you reload (stop/start) the TDS (or any other
>> web app) a number of times without restarting Tomcat. Because of this
>> and other issues, we recommend stop/starting Tomcat when possible (at
>> least occasionally) rather than stop/starting the TDS.
>>
>> We haven't yet thoroughly tested the TDS on Tomcat 7 so we don't
>> recommend switching to Tomcat 7 yet.
>>
>> Ethan
>>
>> [1] http://wiki.apache.org/tomcat/MemoryLeakProtection#customThreadLocal
>>
>> On 4/16/2011 5:03 PM, Roy Mendelssohn wrote:
>>> Hi All:
>>>
>>> Running TDS 4.2.6. Had to do a bunch of start/stops to get the ncISO
>>> services and F-TDS running. On shutdown, I get a string of errors of the
>>> form:
>>>
>>> SEVERE: The web application [/thredds] created a ThreadLocal with
>>> key of type [org.apache.log4j.helpers.ThreadLocalMap] (value
>>> [org.apache.log4j.helpers.ThreadLocalMap@612ef6]) and a value of type
>>> [java.util.Hashtable] (value [{startTime=1302990404971, userid=-,
>>> host=192.168.31.18, ident=-, ID=197, request="GET
>>> /thredds/wms/SODA/2.0.3?VERSION=1.3.0&SERVICE=WMS&REQUEST=GetMap&CRS=CRS:84&WIDTH=400&HEIGHT=200&LAYERS=temp&BBOX=-180.0,-75.25,180.0,89.25&TRANSPARENT=TRUE&STYLES=&FORMAT=image/png
>>> HTTP/1.1"}]) but failed to remove it when the web application was
>>> stopped. This is very likely to create a memory leak.
>>>
>>> Thanks,
>>>
>>> -Roy
>>
>> _______________________________________________
>> thredds mailing list
>> thredds@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>
> Tom Kunicki
> Center for Integrated Data Analytics
> U.S. Geological Survey
> 8505 Research Way
> Middleton, WI 53562
>
> _______________________________________________
> thredds mailing list
> thredds@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
