At some point we will have a streaming server, which would solve the problem.
But not today :(
Heiko Klein wrote:
> Thanks all for your replies,
>
> it is a pity that thredds/opendap cannot be configured to check
> available memory before serving a request. We have enabled the
> http-server, but this won't solve the problem of people pressing the
> 'Get Ascii' button and with that causing a DoS attack, accidentally or not.
>
> On the other hands, this answers my former question concerning
> thredds-administration. I will start looking for a 64bit machine full
> with memory.
>
> Best regards,
>
> Heiko
>
> Ethan Davis wrote:
>> Hi Heiko,
>>
>> Roy is right, the TDS requires a good amount of memory to handle large
>> OPeNDAP data requests. (And the OPeNDAP ASCII requests, in particular,
>> are very big memory hogs.) For downloads of entire files, you might
>> consider enabling straight HTTP access for your data files (more on that
>> below).
>>
>> On a 32-bit machine, there is 2GB of addressable memory. The JVM heap
>> space is limited at closer to 1.5 or 1.6 GB. A 64-bit machine/OS/JVM
>> will definitely give you more room to play.
>>
>> Straight HTTP downloads ...
>>
>> If you don't already have the TDS setup to serve data over HTTP, you
>> will need to modify the configuration catalogs to include the HTTP
>> service. It should look something like the following:
>>
>> <service name="all" serviceType="Compound" base="" >
>> <service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/" />
>> <service name="http" serviceType="HTTPServer"
>> base="/thredds/fileServer/" />
>> </service>
>>
>> with datasets referencing the serviceName "all".
>>
>> More information can be found at
>> http://www.unidata.ucar.edu/projects/THREDDS/tech/tutorial/ConfigCatalogs.html
>>
>> Ethan
>>
>> Roy Mendelssohn wrote:
>>> Yes, but in reality it doesn't appear to work on even that big a
>>> setting. I believe some people have THREDDS working in 64-bit (see
>>> for example
>>> http://www.unidata.ucar.edu/mailing_lists/archives/thredds/2008-March/001008.html)
>>>
>>> , but we do not. Our experience is that is takes a reasonably
>>> larger amount of heap than the file size to deal with it.
>>>
>>> Happy Thanksgiving Nathan,
>>>
>>> -Roy
>>>
>>> On Nov 27, 2008, at 10:01 AM, Nathan Potter wrote:
>>>
>>>> Isn't the JVM heap a capped at 2GB for 32-bit operating systems/JVM's?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Nov 27, 2008, at 7:19 AM, Roy Mendelssohn wrote:
>>>>
>>>>> I do believe THREDDS does all f its work in memory, if so you have
>>>>> to
>>>>> greatly increase the java hep settings. We run at 1500Mb and still
>>>>> run into this. I would be curious to know if even with a very large
>>>>> heap size if file sof tha size can be successfully transferred.
>>>>>
>>>>> -Roy
>>>>>
>>>>> On Nov 27, 2008, at 7:11 AM, Heiko Klein wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I use thredds as opendap server for some large netcdf-files (CF-1.0,
>>>>>> size > 1.1GB). When selecting to download the whole file with the
>>>>>> 'Get
>>>>>> ASCII' or 'GET BINARY', thredds crashes with a 'Java heap error'.
>>>>>>
>>>>>> I'm currently using a default memory setup, i.e. -Xmx512MB for
>>>>>> tomcat.
>>>>>> How much memory will I need to avoid this problem. Does DODS read
>>>>>> the
>>>>>> whole file at once or variable after variable, or even slice after
>>>>>> slice? Or are there some adjustments I can make to reduce memory
>>>>>> consumption?
>>>>>>
>>>>>> Best regards,
>>>>>>
>>>>>> Heiko
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Details:
>>>>>>
>>>>>> DODServlet ERROR (anyExceptionHandler): java.lang.OutOfMemoryError:
>>>>>> Java
>>>>>> heap space
>>>>>> ReqState:
>>>>>> serverClassName: 'thredds.server.opendap.NcDODSServlet'
>>>>>> dataSet: 'XXXX.nc'
>>>>>> requestSuffix: 'ascii'
>>>>>> CE: ''
>>>>>> compressOK: true
>>>>>> InitParameters:
>>>>>> DebugOn: ''
>>>>>> maxNetcdfFilesCached: '10'
>>>>>>
>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>> at ucar.nc2.N3raf.readData(N3raf.java:77)
>>>>>> at ucar.nc2.N3iosp.readData(N3iosp.java:159)
>>>>>> at ucar.nc2.NetcdfFile.readData(NetcdfFile.java:1374)
>>>>>> at ucar.nc2.Variable._read(Variable.java:923)
>>>>>> at ucar.nc2.Variable.read(Variable.java:618)
>>>>>> at thredds.server.opendap.NcSDArray.read(NcSDArray.java:102)
>>>>>> at thredds.server.opendap.NcSDGrid.read(NcSDGrid.java:59)
>>>>>> at opendap.servlet.AsciiWriter.writeAsc(AsciiWriter.java:95)
>>>>>> at opendap.servlet.AsciiWriter.toASCII(AsciiWriter.java:56)
>>>>>> at
>>>>>> opendap.servlet.AbstractServlet.doGetASC(AbstractServlet.java:1029)
>>>>>> at opendap.servlet.AbstractServlet.doGet(AbstractServlet.java:
>>>>>> 1630)
>>>>>> at
>>>>>> thredds.server.opendap.NcDODSServlet.doGet(NcDODSServlet.java:269)
>>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:698)
>>>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
>>>>>> ...
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thredds-Version:
>>>>>> Built-By: caron
>>>>>> Built-On: 2008-02-20 22:56:05
>>>>>> Implementation-Title: THREDDS Data Server
>>>>>> Implementation-Version: 3.16.33
>>>>>> Implementation-Vendor: UCAR/Unidata
>>>>>>
>>>>>> Running on tomcat 6.0.16 with jdk.1.6.0_04 on debian 4.0.
>
