Re: Thredds out of memory

Tennessee Leeuwenburg wrote:

I guessed as much. Can you give me a back-of-the-envelope calculation as to the relationship between request size and memory requirements? Say I want to get the whole 500Mb file. Do I need 500Mb, or 1Gb - 500 for the file and 500 for the DODS object?

im pretty sure it will read it in once into memory and manipulate it from there, so 500 Mb.


On top of that, how much is needed for the rest of thredds to do its business in terms of building the dds / das etc?

this is pretty small, a few 10K and constant (not proportional to data request)


I can imagine streaming would be a medium-hard problem, especially if the netCDF structure is significantly different from the DODS structure, which I suppose could sometimes happen. But it would definately make a difference to the performance of the system I am building...

yeah, as i said it will take a while to get there. also, it will be done in the version 2.2 library.


Cheers,
-T


Its the size of the data request that determines the memory needeed, not the file per se.

unfortunately, we currently have to bring the whole request into memory before sending it out. eventually we will modify both netcdf-java and netcdf-dods to allow data to be streamed. however i doubt we can get to it before the end of the year.

meanwhile your only recourse is to increase java heap space. you could also modify the code to test for data request size and reject anything thats too big.





  • 2005 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdf-java archives: