Re: [thredds] Severe errors on shutdown (TDS 4.2.6.)

Hi Roy and Ethan,

Just for clarity this is a more of a warning to indicated the presence of a 
PermGen leak.  What a PermGen leak means in this context is that the classes 
associated with the TDS war can not be unloaded from PermGen memory.  This is 
of no consequence unless the application is undeployed or redeployed during a 
given runtime of the JVM.  Because the classes for the TDS war cannot be 
unloaded the classes loaded for each deployment stay in memory.  After a 
certain number of deployment cycles PermGen memory will be exhausted and the 
JVM will effectively become unresponsive (classes can't be loaded, strings 
can't be interned, etc).

Longer explanation of this specific error...

The specific message states that a value stored in a ThreadLocal instance is a 
class loaded from the TDS war.  The implementation of ThreadLocal (under the 
JRE hood) uses a static variable in a class loaded by the system class loader 
(present the entire JVM runtime).   Now since this value is loaded by the TDS 
webapp class loader and there is a reference to this value held by a 
ThreadLocal (loaded by the system class loader) and every class instance has a 
reference to it's class loader (and the class loader has a reference to all the 
classes it's loaded) the classes loaded for the TDS deployment can now never be 
unloaded.

The way to avoid this is to *always* call remove when ThreadLocal is used.  
Practically speaking many of these leaks are introduced by dependencies (like 
Log4J) but many of these jars offer clean-up routines.  The error below is 
introduced by Log4J and one should be able to call 
org.apache.log4j.LogManager.shutdown() on servlet unloading to force it to 
release all it's resources.

These are very difficult leaks to track down and they are often (but not 
always) introduced by dependencies.  I would hope some effort would be used to 
track these down and remove them as often times restarting a JVM server on 
redeploy is not an ideal solution.

Tom



On Apr 18, 2011, at 6:19 PM, Ethan Davis wrote:

> Hi Roy,
> 
> Yes, the TDS gets this type of messages on TDS shutdown when running
> under Tomcat 6.0.24 or later. Tomcat looks for memory (thread) leaks
> when web applications are stopped and tries to shut the threads down.
> Tomcat evidently started detecting this particular type of thread
> leakage in Tomcat 6.0.24 but proper shutdown of them isn't available
> until Tomcat 7.0.6 [1]. It does not appear that these threads are leaked
> except upon shutdown.
> 
> This could be an issue if you reload (stop/start) the TDS (or any other
> web app) a number of times without restarting Tomcat. Because of this
> and other issues, we recommend stop/starting Tomcat when possible (at
> least occasionally) rather than stop/starting the TDS.
> 
> We haven't yet thoroughly tested the TDS on Tomcat 7 so we don't
> recommend switching to Tomcat 7 yet.
> 
> Ethan
> 
> [1] http://wiki.apache.org/tomcat/MemoryLeakProtection#customThreadLocal
> 
> On 4/16/2011 5:03 PM, Roy Mendelssohn wrote:
>> Hi All:
>> 
>> Running TDS 4.2.6. Had to do a bunch of start/stops to get the ncISO
>> services and F-TDS running. On shutdown, I get a string of errors of the
>> form:
>> 
>> SEVERE: The web application [/thredds] created a ThreadLocal with
>> key of type [org.apache.log4j.helpers.ThreadLocalMap] (value
>> [org.apache.log4j.helpers.ThreadLocalMap@612ef6]) and a value of type
>> [java.util.Hashtable] (value [{startTime=1302990404971, userid=-,
>> host=192.168.31.18, ident=-, ID=197, request="GET
>> /thredds/wms/SODA/2.0.3?VERSION=1.3.0&SERVICE=WMS&REQUEST=GetMap&CRS=CRS:84&WIDTH=400&HEIGHT=200&LAYERS=temp&BBOX=-180.0,-75.25,180.0,89.25&TRANSPARENT=TRUE&STYLES=&FORMAT=image/png
>> HTTP/1.1"}]) but failed to remove it when the web application was
>> stopped. This is very likely to create a memory leak.
>> 
>> Thanks,
>> 
>> -Roy
> 
> _______________________________________________
> thredds mailing list
> thredds@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe,  visit: 
> http://www.unidata.ucar.edu/mailing_lists/ 

Tom Kunicki
Center for Integrated Data Analytics
U.S. Geological Survey
8505 Research Way
Middleton, WI  53562