Re: [thredds] THREDDS Garbage Collection Tuning

For GC profiling I would also recommend VisualVM along with the VisualGC view 
(might be an optional plugin).  We are using of TDS with the following settings:

"-server -XX:MaxPermSize=512m -XX:+HeapDumpOnOutOfMemoryError 
-XX:HeapDumpPath=$CATALINA_HOME/heapdumps/default -Xmx4096m"


We do have heap dumps on OOME enabled so that we can debug OOME if they do 
occur (they haven't yet with our TB sized aggregations).

GC tuning takes an investment and there isn't a single set of optimal 
parameters.  Even with a single application (like TDS) it really depends on the 
usage patterns of the particular installation.

Some quick notes:

1. The -XX:MaxPermSize is important with TDS, the default is 80m, you want 
something higher.   
2.  Don't be overzealous with -Xmx.  
        2.1 If you are running a 32bit JRE you will be limited to ~1.5G (maybe 
higher with some an OS other than windows).  The JRE won't start if the value 
is too high.
        2.2 There's probably a limit with the 64bit JRE but I haven't found it 
yet.  Be warned that large heaps can lead to large pauses as OldGen GCs can be 
deferred until the heap is exhausted.
        2.3 Make sure the max heap doesn't exceed the amount of physical memory 
along with some room for the OS and other processes.  You don't want to start 
page swapping to disk.
3. If you are willing to watch the GC with VisualVM or Eclipse MAT, watch the 
use of NewGen/Eden Space.  Examine the per request footprint and try to keep 
per-request memory allocations from being pushed into OldGen.  NewGen GC is 
cheap (but gets more expensive as it increases), OldGen GC is expensive and is 
what leads to the observable pauses (this is related to the note in 2.2).  I 
don't know if TDS will benefit from this.  If it would, you would want to 
profile per-request allocations after all the caches have loaded...  You'll 
observe a sawtooth pattern, you want to make the the top of the sawtooth isn't 
clipped and pushed into OldGen.

Many more details here.  
http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.html.

Tom 


On Jun 1, 2011, at 11:53 AM, john caron wrote:

> On 6/1/2011 9:57 AM, Rich Signell wrote:
>> Ethan,
>> 
>> According this this article:
>> http://www.petefreitag.com/item/746.cfm
>> it says
>> 
>> the "OutOfMemoryError - GC overhead limit exceeded" exception is
>> thrown by the garbage collector when it is spending way too much time
>> collecting garbage. This error essentially means that you need to add
>> more memory, or reconfigure your garbage collection arguments.
>> 
>> So it seems like the same issue.    Our settings are:
>> export JAVA_OPTS="-XX:MaxPermSize=256M -Xmx2048m -Xms512m -server
>> -Djava.awt.headless=true"
>> 
>> -Rich
> 
> "OutOfMemoryError - GC overhead limit exceeded" means you dont have enough 
> heap (-Xmx2048m). The MaxPermSize is different and wont directly affect this 
> problem.
> 
> If this is a site with many catalogs, I have an alpha version of TDS that is 
> ready for testing that does not cache catalogs, and will probably fix your 
> problem. If you want to try this out, contact me off list.
> 
> Otherwise, get a memory heap dump and put it on an ftp or web server (they 
> tend to be very large) and send me the URL, so we can see why you are running 
> out of memory. Google "how to get a Java heap dump" to see how. If you want 
> to sink your teeth into this problem, the Eclipse MAT tool is the one to use.
> 
> 
> _______________________________________________
> thredds mailing list
> thredds@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe,  visit: 
> http://www.unidata.ucar.edu/mailing_lists/