Re: [thredds] thredds.filesystem.ControllerOS massive context switching

Hi Brian:

1) on startup, the TDS needs to scan your directories to see whats in the collection. once scanned, it wont do it again.

2) FMRC will be refactored (probably in 4.4) to write the results to disk, and startup=nocheck will use those if available. until then, it has to get scanned

3) i do have an experimental ControllerCache class that will cache the OS file scan. It wasnt reliable enough across platforms for files that were changing, so i disabled. If you want to play with it, contact me. it would be most useful for static collections.

How big is the collection, and how long does it take to scan?

John

On 7/31/2013 9:45 AM, Brian McKenna wrote:
I’m having some issues with THREDDS and a very high number of
interrupts/context switches killing the server’s performance. Increasing
the LOG level shows the high switching related to
thredds.filesystem.ControllerOS. As soon as the messages pasted below
begin, the interrupt/context switch number jumps from <100 to nearly
20,000. CPU and memory do not appear to be a limiting factor, their
loads remain low. The filesystem is mounted via NFS, but read/write
speeds (via dd) seem fine, greater than 150 megabytes/s. Those speeds
are of course without the massive context switching overhead.

Despite <update startup ="nocheck" /> being declared (see catalog
below), ControllerOS seems to be called in a loop somewhere and appears
to be listing every directory and file within and repeats itself once
complete.

Is there a way to avoid such heavy context switching related to this
listing of files, possibly  by consolidating the files to a single
location, or disabling the ‘listing’ being done by ControllerOS all
together?

Thanks,
Brian McKenna

threddsServlet.log

-----------------------

2013-07-31T09:06:53.614 -0400 [     11067][        ] DEBUG -
thredds.filesystem.ControllerOS - List Directory
/data/maracoos/ensemble/WRF_ensemble_small

2013-07-31T09:06:53.630 -0400 [     11083][        ] DEBUG -
thredds.filesystem.ControllerOS -   File
/data/maracoos/ensemble/WRF_ensemble_small/2012.02.10.00

2013-07-31T09:06:53.633 -0400 [     11086][        ] DEBUG -
thredds.filesystem.ControllerOS - List Directory
/data/maracoos/ensemble/WRF_ensemble

2013-07-31T09:06:53.634 -0400 [     11087][        ] DEBUG -
thredds.filesystem.ControllerOS -   File
/data/maracoos/ensemble/WRF_ensemble_small/2012.06.03.00

2013-07-31T09:06:53.638 -0400 [     11091][        ] DEBUG -
thredds.filesystem.ControllerOS -   File
/data/maracoos/ensemble/WRF_ensemble_small/2012.09.30.00

2013-07-31T09:06:53.641 -0400 [     11094][        ] DEBUG -
thredds.filesystem.ControllerOS -   File
/data/maracoos/ensemble/WRF_ensemble_small/2012.03.05.00

2013-07-31T09:06:53.645 -0400 [     11098][        ] DEBUG -
thredds.filesystem.ControllerOS -   File
/data/maracoos/ensemble/WRF_ensemble_small/2012.03.14.12

2013-07-31T09:06:53.648 -0400 [     11101][        ] DEBUG -
thredds.filesystem.ControllerOS -   File
/data/maracoos/ensemble/WRF_ensemble_small/2012.10.05.12

2013-07-31T09:06:53.651 -0400 [     11104][        ] DEBUG -
thredds.filesystem.ControllerOS -   File
/data/maracoos/ensemble/WRF_ensemble_small/2012.10.01.00

2013-07-31T09:06:53.653 -0400 [     11106][        ] DEBUG -
thredds.filesystem.ControllerOS - List Directory
/data/maracoos/ensemble/AKQ_WRF

2013-07-31T09:06:53.659 -0400 [     11112][        ] DEBUG -
thredds.filesystem.ControllerOS -   File
/data/maracoos/ensemble/WRF_ensemble/2012.02.10.00

2013-07-31T09:06:53.660 -0400 [     11113][        ] DEBUG -
thredds.filesystem.ControllerOS -   File
/data/maracoos/ensemble/WRF_ensemble_small/2012.06.01.00

2013-07-31T09:06:53.664 -0400 [     11117][        ] DEBUG -
thredds.filesystem.ControllerOS -   File
/data/maracoos/ensemble/WRF_ensemble/2012.06.03.00

2013-07-31T09:06:53.664 -0400 [     11117][        ] DEBUG -
thredds.filesystem.ControllerOS -   File
/data/maracoos/ensemble/WRF_ensemble_small/2012.11.16.00

2013-07-31T09:06:53.668 -0400 [     11121][        ] DEBUG -
thredds.filesystem.ControllerOS -   File
/data/maracoos/ensemble/WRF_ensemble/2011.12.19.00

2013-07-31T09:06:53.668 -0400 [     11121][        ] DEBUG -
thredds.filesystem.ControllerOS -   File
/data/maracoos/ensemble/WRF_ensemble_small/2012.05.04.12

featureCollection from catalog

---------------------

<featureCollection featureType="FMRC" name="Ensemble WRF (large grid)
Forecast Aggregation Collection" harvest="true"
path="WRF-Large/Agg-Collection.nc">

    <metadata inherited="true">

      <serviceName>agg</serviceName>

      <dataFormat>NetCDF</dataFormat>

    </metadata>

    <collection
spec="/data/maracoos/ensemble/WRF_ensemble/**/#yyyy.MM.dd.HH#.*\.nc$"
name=" WRF_Large_Ensemble" olderThan="5 min" />

    <update startup="nocheck" trigger="allow" />

    <fmrcConfig regularize="true" datasetTypes="Best Files Runs" />

    <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>

      <variable name="lat" orgName="g0_lat_0">

        <attribute name="standard_name" value="latitude" />

      </variable>

      <variable name="lon" orgName="g0_lon_1">

        <attribute name="standard_name" value="longitude" />

      </variable>

      <variable name="u" orgName="U_GRD_GDS0_HTGL_10">

        <attribute name="standard_name" value="eastward_wind_velocity" />

        <attribute name="coordinates" value="time lat lon" />

      </variable>

      <variable name="v" orgName="V_GRD_GDS0_HTGL_10">

        <attribute name="standard_name" value="northward_wind_velocity" />

        <attribute name="coordinates" value="time lat lon" />

      </variable>

      <variable name="pressure" orgName="PRMSL_GDS0_SFC_10">

        <attribute name="coordinates" value="time lat lon" />

      </variable>

      <variable name="temperature" orgName="TMP_GDS0_HTGL_10">

        <attribute name="coordinates" value="time lat lon" />

      </variable>

      <variable name="humidity" orgName="R_H_GDS0_HTGL_10">

        <attribute name="coordinates" value="time lat lon" />

      </variable>

    </netcdf>

  </featureCollection>



_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  visit: 
http://www.unidata.ucar.edu/mailing_lists/




  • 2013 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: