We have aggregations in the 10,000’s that work okay. The frequency of update,
and the time that it takes to update affect the response. We have something
that polls the datasets and therefore the updated aggregation get done usually
before an user makes a request. Perhaps the same code that does the update
should then make a request. Moreover we have found that if the request comes
from the local machine using localhost: it doesn’t time-out and will complete
the request no matter how long it takes.
-Roy
On Mar 13, 2015, at 7:22 PM, John Caron <caron@xxxxxxxx> wrote:
> Hi Rich:
>
> NcML Aggregations should only be used for small collections of files ( a few
> dozen?) , because they are created on-the-fly. FMRC does a better job of
> caching information so things go quicker. It handles the case of a single
> time dimension as a special case of a Forecast model collection. However,
> they too are limited in how much they will scale up, (< 100 ?)
>
> GRIB collection in v4.6.0 finally are almost ready for large scale
> collections (> 10K files). I will be improving the FMRC to use some of the
> techniques in GRIB collections, slated for version 4.6.1. Not yet sure how
> far that will get in scaling up, but I think we can do much better than now.
>
> So how many files and variables are in the HF Radar collection?
>
> John
>
> On Fri, Mar 13, 2015 at 2:50 PM, Signell, Richard <rsignell@xxxxxxxx> wrote:
> Thredds community,
>
> The largest archive of HF Radar ocean surface current data is being
> served by THREDDS at
> http://hfrnet.ucsd.edu/thredds/catalog.html, but the erratic
> performance of the joinExisting aggregations has made them difficult
> to use. The folks at UCSD discovered that if they use FMRC
> aggregations they work much better than the joinExisting, as borne out
> by this Ipython Notebook, where we just request WMS services from the
> two aggregations every minute for one hour:
>
> http://nbviewer.ipython.org/gist/rsignell-usgs/139d5481d74a1181e576
>
> I don't understand this behavior. The joinExisting was designed for
> this type of aggregation (simply joining netcdf files along the time
> dimension) and the FMRC was instead designed for files with
> overlapping forecast times. But there is no arguing with the results
> of this test: FMRC is clearly working better.
>
> Anyone have insight into why we are getting these results?
>
> Are there settings that could be changed to improve the performance of
> the joinExisting aggregation?
>
> Thanks,
> Rich
>
> P.S. the existing aggregation catalog and threddsConfig.xml settings
> are shown at the end of the notebook
>
> --
> Dr. Richard P. Signell (508) 457-2229
> USGS, 384 Woods Hole Rd.
> Woods Hole, MA 02543-1598
>
> _______________________________________________
> thredds mailing list
> thredds@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
>
>
> _______________________________________________
> thredds mailing list
> thredds@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
**********************
"The contents of this message do not reflect any position of the U.S.
Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
***Note new address and phone***
110 Shaffer Road
Santa Cruz, CA 95060
Phone: (831)-420-3666
Fax: (831) 420-3980
e-mail: Roy.Mendelssohn@xxxxxxxx www: http://www.pfeg.noaa.gov/
"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.
