Hi Heiko,
Roy is right, the TDS requires a good amount of memory to handle large
OPeNDAP data requests. (And the OPeNDAP ASCII requests, in particular,
are very big memory hogs.) For downloads of entire files, you might
consider enabling straight HTTP access for your data files (more on that
below).
On a 32-bit machine, there is 2GB of addressable memory. The JVM heap
space is limited at closer to 1.5 or 1.6 GB. A 64-bit machine/OS/JVM
will definitely give you more room to play.
Straight HTTP downloads ...
If you don't already have the TDS setup to serve data over HTTP, you
will need to modify the configuration catalogs to include the HTTP
service. It should look something like the following:
<service name="all" serviceType="Compound" base="" >
<service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/" />
<service name="http" serviceType="HTTPServer"
base="/thredds/fileServer/" />
</service>
with datasets referencing the serviceName "all".
More information can be found at
http://www.unidata.ucar.edu/projects/THREDDS/tech/tutorial/ConfigCatalogs.html
Ethan
Roy Mendelssohn wrote:
> Yes, but in reality it doesn't appear to work on even that big a
> setting. I believe some people have THREDDS working in 64-bit (see
> for example
> http://www.unidata.ucar.edu/mailing_lists/archives/thredds/2008-March/001008.html)
>
> , but we do not. Our experience is that is takes a reasonably
> larger amount of heap than the file size to deal with it.
>
> Happy Thanksgiving Nathan,
>
> -Roy
>
> On Nov 27, 2008, at 10:01 AM, Nathan Potter wrote:
>
>>
>> Isn't the JVM heap a capped at 2GB for 32-bit operating systems/JVM's?
>>
>>
>>
>>
>>
>>
>> On Nov 27, 2008, at 7:19 AM, Roy Mendelssohn wrote:
>>
>>> I do believe THREDDS does all f its work in memory, if so you have
>>> to
>>> greatly increase the java hep settings. We run at 1500Mb and still
>>> run into this. I would be curious to know if even with a very large
>>> heap size if file sof tha size can be successfully transferred.
>>>
>>> -Roy
>>>
>>> On Nov 27, 2008, at 7:11 AM, Heiko Klein wrote:
>>>
>>>> Hi,
>>>>
>>>> I use thredds as opendap server for some large netcdf-files (CF-1.0,
>>>> size > 1.1GB). When selecting to download the whole file with the
>>>> 'Get
>>>> ASCII' or 'GET BINARY', thredds crashes with a 'Java heap error'.
>>>>
>>>> I'm currently using a default memory setup, i.e. -Xmx512MB for
>>>> tomcat.
>>>> How much memory will I need to avoid this problem. Does DODS read
>>>> the
>>>> whole file at once or variable after variable, or even slice after
>>>> slice? Or are there some adjustments I can make to reduce memory
>>>> consumption?
>>>>
>>>> Best regards,
>>>>
>>>> Heiko
>>>>
>>>>
>>>>
>>>>
>>>> Details:
>>>>
>>>> DODServlet ERROR (anyExceptionHandler): java.lang.OutOfMemoryError:
>>>> Java
>>>> heap space
>>>> ReqState:
>>>> serverClassName: 'thredds.server.opendap.NcDODSServlet'
>>>> dataSet: 'XXXX.nc'
>>>> requestSuffix: 'ascii'
>>>> CE: ''
>>>> compressOK: true
>>>> InitParameters:
>>>> DebugOn: ''
>>>> maxNetcdfFilesCached: '10'
>>>>
>>>> java.lang.OutOfMemoryError: Java heap space
>>>> at ucar.nc2.N3raf.readData(N3raf.java:77)
>>>> at ucar.nc2.N3iosp.readData(N3iosp.java:159)
>>>> at ucar.nc2.NetcdfFile.readData(NetcdfFile.java:1374)
>>>> at ucar.nc2.Variable._read(Variable.java:923)
>>>> at ucar.nc2.Variable.read(Variable.java:618)
>>>> at thredds.server.opendap.NcSDArray.read(NcSDArray.java:102)
>>>> at thredds.server.opendap.NcSDGrid.read(NcSDGrid.java:59)
>>>> at opendap.servlet.AsciiWriter.writeAsc(AsciiWriter.java:95)
>>>> at opendap.servlet.AsciiWriter.toASCII(AsciiWriter.java:56)
>>>> at
>>>> opendap.servlet.AbstractServlet.doGetASC(AbstractServlet.java:1029)
>>>> at opendap.servlet.AbstractServlet.doGet(AbstractServlet.java:
>>>> 1630)
>>>> at
>>>> thredds.server.opendap.NcDODSServlet.doGet(NcDODSServlet.java:269)
>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:698)
>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>>>> ...
>>>>
>>>>
>>>>
>>>> Thredds-Version:
>>>> Built-By: caron
>>>> Built-On: 2008-02-20 22:56:05
>>>> Implementation-Title: THREDDS Data Server
>>>> Implementation-Version: 3.16.33
>>>> Implementation-Vendor: UCAR/Unidata
>>>>
>>>> Running on tomcat 6.0.16 with jdk.1.6.0_04 on debian 4.0.
--
Ethan R. Davis Telephone: (303) 497-8155
Software Engineer Fax: (303) 497-8690
UCAR Unidata Program Center E-mail: edavis@xxxxxxxx
P.O. Box 3000
Boulder, CO 80307-3000 http://www.unidata.ucar.edu/
---------------------------------------------------------------------------
