Re: [thredds] joinExisting and FMRC aggregation performance

  • To: John Caron <caron@xxxxxxxx>
  • Subject: Re: [thredds] joinExisting and FMRC aggregation performance
  • From: "Signell, Richard" <rsignell@xxxxxxxx>
  • Date: Sat, 14 Mar 2015 12:47:58 -0400
John,

> NcML Aggregations should only be used for small collections of files ( a few
> dozen?) , because they are created on-the-fly.

The HFRADAR data is using a joinExisting aggregation in a THREDDS
catalog.   Is that what you are calling NcML aggregation?
I was thinking that NcML aggregation referred to the practice of
writing an NcML file and dropping that into a folder along with the
data files where it can be picked up by a DatasetScan.

> FMRC does a better job of
> caching information so things go quicker. It handles the case of a single
> time dimension as a special case of a Forecast model collection. However,
> they too are limited in how much they will scale up, (< 100 ?)
>
> So how many files and variables are in the HF Radar collection?

There are currently 27,986 NetCDF files in the aggregation, each with
a single time record containing the HF radar data for the hour.    It
seems that the FMRC is handling this just fine, with reliable WMS
response times of about one second.

As Dave Blodgett points out, a better approach here might be to
periodically combine a bunch of these hourly files into, say, monthly
files, which would result in higher performance, less utilization of
disk space, and quicker aggregation.

I still don't understand what is happening with the joinExisting
aggregation, however -- why it periodically (but not regularly) takes
50 seconds or more to respond.

-- 
Dr. Richard P. Signell   (508) 457-2229
USGS, 384 Woods Hole Rd.
Woods Hole, MA 02543-1598



  • 2015 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: