[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [thredds] "Too Many Open Files" Error. Dataset too big?



Hi Michael:

You should set NetcdfFileCache to be at least 100 or so less than ulimit, to allow other files to be open.

All agg files need to be touched once, but not simultaneously; they are opened and then closed.

John

On 10/25/2013 12:20 PM, Michael McDonald wrote:
John et al,

Aggregations are not supposed to leave files open. If they are, thats a bug.
The number of open files should typically be less than 1000, no matter how big 
your aggregations are. It is dependent on the file cache sizes, see:

We set our NetcdfFileCache to match what we set our ulimit max # of open files to (1mil). The hope of this is that as many files as possible are cached in memory by the server, for quicker catalog scans/requests. We never even approach 1mil concurrently open files.

    <NetcdfFileCache>
     <minFiles>1000</minFiles>
     <maxFiles>1000000</maxFiles>
     <scour>24 hours</scour>
   </NetcdfFileCache>


It was also my understanding that ALL of the files in a particular dataset scan need to be touched/opened/checked before being folded into an aggregation (this is upon initial scan for saving into the cache/agg folder). After this initial indexing, the average # of open files will drop off since queries are only picking/choosing a few files from this catalog, but you still need to have all the proper system limits bumped up in order for this initial scan/cache to ever successfully complete in a timely fashion. correct/incorrect?

/mike


_______________________________________________
thredds mailing list
address@hidden
For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/



NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.