Re: [thredds] Java heap problems with large ASCII downloads



Isn't the JVM heap a capped at 2GB for 32-bit operating systems/JVM's?






On Nov 27, 2008, at 7:19 AM, Roy Mendelssohn wrote:

I do believe THREDDS does all f its work in memory,  if so you have to
greatly increase the java hep settings.  We run at 1500Mb and still
run into this.  I  would be curious to know if even with a very large
heap size if file sof tha size can be successfully transferred.

-Roy

On Nov 27, 2008, at 7:11 AM, Heiko Klein wrote:

Hi,

I use thredds as opendap server for some large netcdf-files (CF-1.0,
size >  1.1GB). When selecting to download the whole file  with the
'Get
ASCII' or 'GET BINARY', thredds crashes with a 'Java heap error'.

I'm currently using a default memory setup, i.e. -Xmx512MB for tomcat.
How much memory will I need to avoid this problem. Does DODS read the
whole file at once or variable after variable, or even slice after
slice? Or are there some adjustments I can make to reduce memory
consumption?

Best regards,

Heiko




Details:

DODServlet ERROR (anyExceptionHandler): java.lang.OutOfMemoryError:
Java
heap space
ReqState:
serverClassName:    'thredds.server.opendap.NcDODSServlet'
dataSet:            'XXXX.nc'
requestSuffix:      'ascii'
CE:                 ''
compressOK:          true
InitParameters:
  DebugOn: ''
  maxNetcdfFilesCached: '10'

java.lang.OutOfMemoryError: Java heap space
      at ucar.nc2.N3raf.readData(N3raf.java:77)
      at ucar.nc2.N3iosp.readData(N3iosp.java:159)
      at ucar.nc2.NetcdfFile.readData(NetcdfFile.java:1374)
      at ucar.nc2.Variable._read(Variable.java:923)
      at ucar.nc2.Variable.read(Variable.java:618)
      at thredds.server.opendap.NcSDArray.read(NcSDArray.java:102)
      at thredds.server.opendap.NcSDGrid.read(NcSDGrid.java:59)
      at opendap.servlet.AsciiWriter.writeAsc(AsciiWriter.java:95)
      at opendap.servlet.AsciiWriter.toASCII(AsciiWriter.java:56)
      at
opendap.servlet.AbstractServlet.doGetASC(AbstractServlet.java:1029)
      at opendap.servlet.AbstractServlet.doGet(AbstractServlet.java:
1630)
      at
thredds.server.opendap.NcDODSServlet.doGet(NcDODSServlet.java:269)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:698)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
...



Thredds-Version:
Built-By: caron
Built-On: 2008-02-20 22:56:05
Implementation-Title: THREDDS Data Server
Implementation-Version: 3.16.33
Implementation-Vendor: UCAR/Unidata

Running on tomcat 6.0.16 with jdk.1.6.0_04 on debian 4.0.

--
Dr. Heiko Klein                              Tel. + 47 22 96 32 58
Development Section / IT Department          Fax. + 47 22 69 63 55
Norwegian Meteorological Institute           http://www.met.no
P.O. Box 43 Blindern  0313 Oslo NORWAY
_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  visit: 
http://www.unidata.ucar.edu/mailing_lists/

**********************
"The contents of this message do not reflect any position of the U.S.
Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"

_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  visit: 
http://www.unidata.ucar.edu/mailing_lists/

= = =
Nathan Potter                        ndp at opendap.org
OPeNDAP, Inc.                        541.752.1852




  • 2008 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: