Hi Ken:
Ken Tanaka wrote:
> John and Roy,
>
> It looks like the current Xmx memory setting is 256 MB for the tomcat
> process. I'll arrange to have the memory increased, but we were
> concerned that we would still run out of memory--just at a higher number
> of files. Earlier posts indicated that I should be able to aggregate a
> large number of files (over 2000), so I wasn't sure if I was configuring
> the server correctly. Should I delve into caching or types of
> performance tuning? We would like to add a file once a day and to an
> unlimited Time coordinate, our joinExisting dimension, indefinitely.
Im currently testing with 20,000 files, which it is handling ok. There is a
small amount of memory retained for each file, which i am currently trying to
minimize.
I think you'll find that if you can give the server 1M or more, your case will
work fine. You will also see significant performance by using java 1.6.
Meanwhile, i will try to keep things as sleek as possible. We will eventually
get object caching working, to ensure there will be no memory limits at all.
New development is being done only on the 4.0 branch. Its stable enough to
start testing with it if you want to compare performance, etc. Look for it on
the TDS home page:
http://www.unidata.ucar.edu/projects/THREDDS/tech/TDS.html
>
> We were thinking of moving a large number of Global Attributes
> (metadata) out of the files and into an NcML file since most of it is
> the same (constant) in all the files. We felt that it made sense to
> place the aggregation and attributes into the NcML file rather than
> clutter the catalog.xml in this case. Will this save memory to abstract
> out the repeated Global Attributes and provide them in a single NcML
> wrapper?
There wont be any real memory savings by factoring out attributes into NcML.
There will be some minor speedup if you actually remove them from the netcdf
files. For now, I wouldn't try to do such optimizations.
Let me know if your problem goes away with a larger memory size.
