Due to the current gap in continued funding from the U.S. National Science Foundation (NSF), the NSF Unidata Program Center has temporarily paused most operations. See NSF Unidata Pause in Most Operations for details.

Re: [thredds] Java heap problems with large ASCII downloads

I do believe THREDDS does all f its work in memory, if so you have to greatly increase the java hep settings. We run at 1500Mb and still run into this. I would be curious to know if even with a very large heap size if file sof tha size can be successfully transferred. 

-Roy

On Nov 27, 2008, at 7:11 AM, Heiko Klein wrote:


Hi,

I use thredds as opendap server for some large netcdf-files (CF-1.0,
size > 1.1GB). When selecting to download the whole file with the 'Get 
ASCII' or 'GET BINARY', thredds crashes with a 'Java heap error'.

I'm currently using a default memory setup, i.e. -Xmx512MB for tomcat.
How much memory will I need to avoid this problem. Does DODS read the
whole file at once or variable after variable, or even slice after
slice? Or are there some adjustments I can make to reduce memory
consumption?

Best regards,

Heiko




Details:
DODServlet ERROR (anyExceptionHandler): java.lang.OutOfMemoryError: Java 
heap space
ReqState:
 serverClassName:    'thredds.server.opendap.NcDODSServlet'
 dataSet:            'XXXX.nc'
 requestSuffix:      'ascii'
 CE:                 ''
 compressOK:          true
 InitParameters:
   DebugOn: ''
   maxNetcdfFilesCached: '10'

java.lang.OutOfMemoryError: Java heap space
       at ucar.nc2.N3raf.readData(N3raf.java:77)
       at ucar.nc2.N3iosp.readData(N3iosp.java:159)
       at ucar.nc2.NetcdfFile.readData(NetcdfFile.java:1374)
       at ucar.nc2.Variable._read(Variable.java:923)
       at ucar.nc2.Variable.read(Variable.java:618)
       at thredds.server.opendap.NcSDArray.read(NcSDArray.java:102)
       at thredds.server.opendap.NcSDGrid.read(NcSDGrid.java:59)
       at opendap.servlet.AsciiWriter.writeAsc(AsciiWriter.java:95)
       at opendap.servlet.AsciiWriter.toASCII(AsciiWriter.java:56)
       at
opendap.servlet.AbstractServlet.doGetASC(AbstractServlet.java:1029)
at opendap.servlet.AbstractServlet.doGet(AbstractServlet.java: 1630) 
       at
thredds.server.opendap.NcDODSServlet.doGet(NcDODSServlet.java:269)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:698)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
...



Thredds-Version:
Built-By: caron
Built-On: 2008-02-20 22:56:05
Implementation-Title: THREDDS Data Server
Implementation-Version: 3.16.33
Implementation-Vendor: UCAR/Unidata

Running on tomcat 6.0.16 with jdk.1.6.0_04 on debian 4.0.

--
Dr. Heiko Klein                              Tel. + 47 22 96 32 58
Development Section / IT Department          Fax. + 47 22 69 63 55
Norwegian Meteorological Institute           http://www.met.no
P.O. Box 43 Blindern  0313 Oslo NORWAY
_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  visit: 
http://www.unidata.ucar.edu/mailing_lists/


**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA." 
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: Roy.Mendelssohn@xxxxxxxx (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
  • 2008 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: