Re: [thredds] Java heap problems with large ASCII downloads

At some point we will have a streaming server, which would solve the problem. 
But not today :(

Heiko Klein wrote:
> Thanks all for your replies,
> 
> it is a pity that thredds/opendap cannot be configured to check 
> available memory before serving a request. We have enabled the 
> http-server, but this won't solve the problem of people pressing the 
> 'Get Ascii' button and with that causing a DoS attack, accidentally or not.
> 
> On the other hands, this answers my former question concerning 
> thredds-administration. I will start looking for a 64bit machine full 
> with memory.
> 
> Best regards,
> 
> Heiko
> 
> Ethan Davis wrote:
>> Hi Heiko,
>>
>> Roy is right, the TDS requires a good amount of memory to handle large
>> OPeNDAP data requests. (And the OPeNDAP ASCII requests, in particular,
>> are very big memory hogs.) For downloads of entire files, you might
>> consider enabling straight HTTP access for your data files (more on that
>> below).
>>
>> On a 32-bit machine, there is 2GB of addressable memory. The JVM heap
>> space is limited at closer to 1.5 or 1.6 GB. A 64-bit machine/OS/JVM
>> will definitely give you more room to play.
>>
>> Straight HTTP downloads ...
>>
>> If you don't already have the TDS setup to serve data over HTTP, you
>> will need to modify the configuration catalogs to include the HTTP
>> service. It should look something like the following:
>>
>> <service name="all" serviceType="Compound" base="" >
>>     <service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/" />
>>     <service name="http" serviceType="HTTPServer"
>> base="/thredds/fileServer/" />
>> </service>
>>
>> with datasets referencing the serviceName "all".
>>
>> More information can be found at
>> http://www.unidata.ucar.edu/projects/THREDDS/tech/tutorial/ConfigCatalogs.html
>>
>> Ethan
>>
>> Roy Mendelssohn wrote:
>>> Yes, but in reality it doesn't appear to work on even that big a  
>>> setting.  I believe some people have THREDDS working in 64-bit  (see  
>>> for example 
>>> http://www.unidata.ucar.edu/mailing_lists/archives/thredds/2008-March/001008.html)
>>>  
>>>   ,  but we do not.  Our experience is that is takes a reasonably  
>>> larger amount of heap than the file size to deal with it.
>>>
>>> Happy Thanksgiving Nathan,
>>>
>>> -Roy
>>>
>>> On Nov 27, 2008, at 10:01 AM, Nathan Potter wrote:
>>>
>>>> Isn't the JVM heap a capped at 2GB for 32-bit operating systems/JVM's?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Nov 27, 2008, at 7:19 AM, Roy Mendelssohn wrote:
>>>>
>>>>> I do believe THREDDS does all f its work in memory,  if so you have  
>>>>> to
>>>>> greatly increase the java hep settings.  We run at 1500Mb and still
>>>>> run into this.  I  would be curious to know if even with a very large
>>>>> heap size if file sof tha size can be successfully transferred.
>>>>>
>>>>> -Roy
>>>>>
>>>>> On Nov 27, 2008, at 7:11 AM, Heiko Klein wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I use thredds as opendap server for some large netcdf-files (CF-1.0,
>>>>>> size >  1.1GB). When selecting to download the whole file  with the
>>>>>> 'Get
>>>>>> ASCII' or 'GET BINARY', thredds crashes with a 'Java heap error'.
>>>>>>
>>>>>> I'm currently using a default memory setup, i.e. -Xmx512MB for  
>>>>>> tomcat.
>>>>>> How much memory will I need to avoid this problem. Does DODS read  
>>>>>> the
>>>>>> whole file at once or variable after variable, or even slice after
>>>>>> slice? Or are there some adjustments I can make to reduce memory
>>>>>> consumption?
>>>>>>
>>>>>> Best regards,
>>>>>>
>>>>>> Heiko
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Details:
>>>>>>
>>>>>> DODServlet ERROR (anyExceptionHandler): java.lang.OutOfMemoryError:
>>>>>> Java
>>>>>> heap space
>>>>>> ReqState:
>>>>>> serverClassName:    'thredds.server.opendap.NcDODSServlet'
>>>>>> dataSet:            'XXXX.nc'
>>>>>> requestSuffix:      'ascii'
>>>>>> CE:                 ''
>>>>>> compressOK:          true
>>>>>> InitParameters:
>>>>>>  DebugOn: ''
>>>>>>  maxNetcdfFilesCached: '10'
>>>>>>
>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>      at ucar.nc2.N3raf.readData(N3raf.java:77)
>>>>>>      at ucar.nc2.N3iosp.readData(N3iosp.java:159)
>>>>>>      at ucar.nc2.NetcdfFile.readData(NetcdfFile.java:1374)
>>>>>>      at ucar.nc2.Variable._read(Variable.java:923)
>>>>>>      at ucar.nc2.Variable.read(Variable.java:618)
>>>>>>      at thredds.server.opendap.NcSDArray.read(NcSDArray.java:102)
>>>>>>      at thredds.server.opendap.NcSDGrid.read(NcSDGrid.java:59)
>>>>>>      at opendap.servlet.AsciiWriter.writeAsc(AsciiWriter.java:95)
>>>>>>      at opendap.servlet.AsciiWriter.toASCII(AsciiWriter.java:56)
>>>>>>      at
>>>>>> opendap.servlet.AbstractServlet.doGetASC(AbstractServlet.java:1029)
>>>>>>      at opendap.servlet.AbstractServlet.doGet(AbstractServlet.java:
>>>>>> 1630)
>>>>>>      at
>>>>>> thredds.server.opendap.NcDODSServlet.doGet(NcDODSServlet.java:269)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:698)
>>>>>>      at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>>>>>> ...
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thredds-Version:
>>>>>> Built-By: caron
>>>>>> Built-On: 2008-02-20 22:56:05
>>>>>> Implementation-Title: THREDDS Data Server
>>>>>> Implementation-Version: 3.16.33
>>>>>> Implementation-Vendor: UCAR/Unidata
>>>>>>
>>>>>> Running on tomcat 6.0.16 with jdk.1.6.0_04 on debian 4.0.
> 


  • 2008 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: