Re: [thredds] "Too Many Open Files" Error. Dataset too big?

Hi all:

Aggregations arent supposed to leave files open. If they are, thats a bug.

The number of open files should typically be less than 1000, no matter how big 
your aggregations are. It is dependent on the file cache sizes, see:

  
http://www.unidata.ucar.edu/software/thredds/v4.4/tds/reference/ThreddsConfigXMLFile.html#FileCache

So if you are seeing lots of open files, we have a file leak i think. You can 
verify by turning your file cache to 0 or very low (10?). Then watch the pfiles 
output to see whats happening.

John



On 10/24/2013 3:15 PM, Kevin Manross wrote:
> 
> This is very helpful Mike - thank you very much for the insight and your 
> experience.
> 
> This will be useful information for requesting the ulimit increase from 
> our sysadmins!
> 
> -kevin.
> 
> 
> On 10/24/13 3:09 PM, Michael McDonald wrote:
>> Kevin,
>> We needed to add a script to "/etc/profile.d" that increases the stock
>> CentOS/RHEL 6.x number of open files. Change $USER == "tomcat" to
>> whatever you run tomcat/apache as (see below). Also assumes that the
>> user is running the sh/bash shell.
>>
>> # cat /etc/profile.d/thredds.sh
>> export TOMCAT_HOME="/usr/share/tomcat6"
>> if [ $USER == "tomcat" ]; then ulimit -n 1000000; fi
>>
>>
>> Our THREDDS servers providing only OPENDAP requests are averaging 40k
>> open files. Our THREDDS servers providing only NCSS requests are
>> averaging 15k open files. We have a lot of files in some of our hycom
>> datasets.
>>
>> /mike
>>
>> On Thu, Oct 24, 2013 at 1:50 PM, Kevin Manross<manross@xxxxxxxx>  wrote:
>>> Hi there,
>>>
>>> I am trying to aggregate netcdf files for some rather large datasets with
>>> tomcat6.0.36/TDS4.19 - 20131008.1358 and am running into a "Too many open
>>> files" error.
>>>
>>> My catalog scan is:
>>>
>>>    <dataset name="ASR 30 km 2d surface analysis"
>>>             ID="ds631.0_anl_2d-AGG"
>>>             urlPath="ds631.0/9/best"
>>>             harvest="true">
>>>       <serviceName>all</serviceName>
>>>       <netcdf 
>>> xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
>>>         <aggregation dimName="time" type="Union">
>>>           <scan location="/glade/p/rda/data/ds631.0/asr30km.anl.2d/"
>>> suffix=".nc" subdirs="false"/>
>>>         </aggregation>
>>>       </netcdf>
>>>     </dataset>
>>>
>>> which contains about 4020 files and totals 470G.
>>>
>>> The message that the server sends when trying to access the OPeNDAP service
>>> is:
>>>
>>>
>>> Error {
>>>      code = 404;
>>>      message =
>>> "/glade/p/rda/data/ds631.0/asr30km.anl.2d/asr30km.anl.2d.20010619.nc (Too
>>> many open files)";
>>> };
>>>
>>>
>>>
>>> When trying the CdmRemote service catalina.out repeatedly reads:
>>>
>>> Oct 24, 2013 11:20:39 AM org.apache.jk.common.ChannelSocket
>>> acceptConnections
>>> WARNING: Exception executing accept
>>> java.net.SocketException: Too many open files
>>>      at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>      at
>>> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:375)
>>>      at java.net.ServerSocket.implAccept(ServerSocket.java:478)
>>>      at java.net.ServerSocket.accept(ServerSocket.java:446)
>>>      at org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:311)
>>>      at
>>> org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:668)
>>>      at
>>> org.apache.jk.common.ChannelSocket$SocketAcceptor.runIt(ChannelSocket.java:879)
>>>      at
>>> org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
>>>      at java.lang.Thread.run(Thread.java:679)
>>>
>>>
>>> The cache settings in my threddsConfig.xml are the default settings.
>>>
>>> Have I overlooked something?
>>>
>>> -kevin.
>>>
>>>
>>> --
>>> Kevin Manross
>>> NCAR/CISL/Data Support Section
>>> Phone: (303)-497-1218
>>> Email:manross@xxxxxxxx
>>> Web:http://rda.ucar.edu
>>>
>>> _______________________________________________
>>> thredds mailing list
>>> thredds@xxxxxxxxxxxxxxxx
>>> For list information or to unsubscribe,  visit:
>>> http://www.unidata.ucar.edu/mailing_lists/
> 
> -- 
> Kevin Manross
> NCAR/CISL/Data Support Section
> Phone: (303)-497-1218
> Email:manross@xxxxxxxx <mailto:manross@xxxxxxxx>
> Web:http://rda.ucar.edu
> 
> 
> _______________________________________________
> thredds mailing list
> thredds@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe,  visit: 
> http://www.unidata.ucar.edu/mailing_lists/
> 



  • 2013 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: