Re: [thredds] ncml aggregation 200 times slower in thredds 4.6.1

  • To: John Caron <caron@xxxxxxxx>
  • Subject: Re: [thredds] ncml aggregation 200 times slower in thredds 4.6.1
  • From: Heiko Klein <heiko.klein@xxxxxx>
  • Date: Wed, 20 May 2015 16:25:50 +0000 (UTC)
Hi John,

in 4.6.1, request-time stays at ~20s each time I try it. Only in 4.3.23 I see a 
huge perfomance-gain (from 7s to 0.1s) after the first fetch.

Heiko

----- Original Message -----
> Hi Heiko:
> 
> Can you see whether the second time you access the dataset, if the times
> are fast again?
> 
> thanks,
> John
> 
> On Wed, May 20, 2015 at 2:07 AM, Heiko Klein <Heiko.Klein@xxxxxx> wrote:
> 
> > Hi,
> >
> > I have some performance problems after upgrading to thredds 4.6.1.
> >
> > I'm aggregating a large dataset with a joinExisting aggregation. Reading
> > the metadata from the aggregation took, with thredds 4.3.23, about 0.1s
> > (first time up to 7s). After upgrading to 4.6.1, the same request takes 20s
> > (second time, first time not measured) and is unusable slow. A 'ncview' of
> > the aggregated dataset is no longer possible.
> >
> > Suspecting some caching problems, I followed the guidelines in
> > http://www.unidata.ucar.edu/software/thredds/current/tds/reference/ThreddsConfigXMLFile.html
> > The aggregation-cache contains two files
> >
> > $ ls -l file-lustre-mnt-heikok-FAUNA-mbr000_all.ncml
> > lustre/mnt/heikok/FAUNA/mbr000_all.ncml
> > -rw-r--r-- 1 tomcat7 tomcat7 74339 May 20 09:43
> > file-lustre-mnt-heikok-FAUNA-mbr000_all.ncml
> > -rw-r--r-- 1 tomcat7 tomcat7 74131 May 20 10:01
> > lustre/mnt/heikok/FAUNA/mbr000_all.ncml
> >
> > (I guess the first one is thredds 4.3, while the second one is thredds 4.6)
> >
> >
> > The ncml-file is:
> >
> > <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
> >   <aggregation dimName="time" type="joinExisting">
> >     <scan location="." regExp=".*snapMet000\.nc$" subdirs="true"/>
> >   </aggregation>
> > </netcdf>
> >
> >
> > It is aggregating 4575 files, each about 1.3G.
> >
> >
> > The requests according to the logs are:
> > 2015-05-20T09:49:17.111 +0200 [    372240][     281] INFO  -
> > threddsServlet - Remote host: 157.249.113.42 - Request: "GET
> > /thredds/dodsC/lustreMnt/heikok/FAUNA/mbr000_all.ncml.dds HTTP/1.1"
> > 2015-05-20T09:49:17.116 +0200 [    372245][     281] INFO  -
> > threddsServlet - Request Completed - 200 - -1 - 5
> > 2015-05-20T09:49:17.117 +0200 [    372246][     282] INFO  -
> > threddsServlet - Remote host: 157.249.113.42 - Request: "GET
> > /thredds/dodsC/lustreMnt/heikok/FAUNA/mbr000_all.ncml.das HTTP/1.1"
> > 2015-05-20T09:49:17.131 +0200 [    372260][     282] INFO  -
> > threddsServlet - Request Completed - 200 - -1 - 14
> > 2015-05-20T09:49:17.134 +0200 [    372263][     283] INFO  -
> > threddsServlet - Remote host: 157.249.113.42 - Request: "GET
> > /thredds/dodsC/lustreMnt/heikok/FAUNA/mbr000_all.ncml.dds HTTP/1.1"
> > 2015-05-20T09:49:17.138 +0200 [    372267][     283] INFO  -
> > threddsServlet - Request Completed - 200 - -1 - 4
> >
> >
> > Best regards,
> >
> > Heiko
> >
> >
> > --
> > Dr. Heiko Klein                   Norwegian Meteorological Institute
> > Tel. + 47 22 96 32 58             P.O. Box 43 Blindern
> > http://www.met.no                 0313 Oslo NORWAY
> >
> > _______________________________________________
> > thredds mailing list
> > thredds@xxxxxxxxxxxxxxxx
> > For list information or to unsubscribe,  visit:
> > http://www.unidata.ucar.edu/mailing_lists/
> >
> 



  • 2015 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: