Re: THREDDS and grib

[Kevin O'Brien <Kevin.M.O'Brien@xxxxxxxx>]

Sounds good.  Let me know if I can help.............

John Caron wrote:
> these are all good questions - there have been similar reports of the 
> agg cache not working like it should. i will have to reproduce to see 
> whats happening.
>
> Kevin O'Brien wrote:
> > Hi John -
> >
> > I tried what you suggested and it didn't seem to have a significant 
> > effect in making the initial access of the aggregated dataset 
> > quicker.  It still took over a minute and a half to open the 
> > dataset.  I've pasted the xml config that I used to define the new 
> > aggregation below.  To be honest, I'm actually kind of glad because I 
> > wasn't looking forward to modifying the guts of the application which 
> > generates the xml config automatically.... :-)
> >
> > I guess I can understand and probably even accept the fact that for 
> > the first time the dataset is accessed, things will be a little 
> > slow.  After that, I presume the dataset is available in the cache, 
> > and of course subsequent accesses prove that it is because the 
> > response is quite quick.    However, if the tomcat server is 
> > restarted, it seems like whatever is in the cache is ignored and the 
> > cache entries have to be rebuilt.   I have my aggregation cache set 
> > like so:
> >
> >  <AggregationCache>
> >    
> > <dir>/home/pmel/DataPortal/apache-tomcat-5.5.25/content/thredds/cacheAged/</dir> 
> >
> >    <scour>24 hours</scour>
> >    <maxAge>90 days</maxAge>
> >  </AggregationCache>  
> > Does that seem correct? Also, as an aside, you mention that you 
> > thought this would be quicker because it avoids the OPeNDAP 
> > URL's....Shouldn't there be some client side caching done w/ the 
> > OPeNDAP datasets?  For example, if I access a remote dataset with 
> > ncdump (or Ferret), and my OPeNDAP caching is turned on my ~/.dodsrc 
> > file, it will cache the response in the ~/.dods_cache directory.  
> > Does any of that happen when OPeNDAP URL's are accessed through TDS???
> >
> > Anyway - here's the xml config I used as per your suggestion:
> >
> >    <dataset ID="CM2.1U-D4_1PctTo2X_I1 atmos daily all vars 
> > 00010101-02201231_2" name="CM2.1U-D4_1PctTo2X_I1 atmos daily all vars 
> > 00010101-02201231_2" 
> > urlPath="ipcc_ar4_CM2.1_R1_1to2x-1_daily_atmos_00010101-02201231_2">
> >        <serviceName>thisDODS3</serviceName>
> >        <netcdf 
> > xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
> >          <aggregation type="union">
> >             <netcdf 
> > xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
> >               <aggregation dimName="time" type="joinExisting">
> >                <netcdf 
> > location="file:/data/gfdl_cm2_1/CM2.1U-D4_1PctTo2X_I1/pp/atmos/ts/daily/pr_A2.00010101-01001231.nc" 
> > ncoords="36500" />
> >                <netcdf 
> > location="file:/data/gfdl_cm2_1/CM2.1U-D4_1PctTo2X_I1/pp/atmos/ts/daily/pr_A2.01010101-02001231.nc" 
> > ncoords="36500" />
> >                <netcdf 
> > location="file:/data/gfdl_cm2_1/CM2.1U-D4_1PctTo2X_I1/pp/atmos/ts/daily/pr_A2.02010101-02201231.nc" 
> > ncoords="7300" />
> >               </aggregation>
> >             </netcdf>
> >             <netcdf 
> > xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
> >               <aggregation dimName="time" type="joinExisting">
> >                <netcdf 
> > location="file:/data/gfdl_cm2_1/CM2.1U-D4_1PctTo2X_I1/pp/atmos/ts/daily/tasmax_A2.00010101-01001231.nc" 
> > ncoords="36500" />
> >                <netcdf 
> > location="file:/data/gfdl_cm2_1/CM2.1U-D4_1PctTo2X_I1/pp/atmos/ts/daily/tasmax_A2.01010101-02001231.nc" 
> > ncoords="36500" />
> >                <netcdf 
> > location="file:/data/gfdl_cm2_1/CM2.1U-D4_1PctTo2X_I1/pp/atmos/ts/daily/tasmax_A2.02010101-02201231.nc" 
> > ncoords="7300" />
> >               </aggregation>
> >             </netcdf>
> >             <netcdf 
> > xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
> >               <aggregation dimName="time" type="joinExisting">
> >                <netcdf 
> > location="file:/data/gfdl_cm2_1/CM2.1U-D4_1PctTo2X_I1/pp/atmos/ts/daily/tasmin_A2.00010101-01001231.nc" 
> > ncoords="36500" />
> >                <netcdf 
> > location="file:/data/gfdl_cm2_1/CM2.1U-D4_1PctTo2X_I1/pp/atmos/ts/daily/tasmin_A2.01010101-02001231.nc" 
> > ncoords="36500" />
> >                <netcdf 
> > location="file:/data/gfdl_cm2_1/CM2.1U-D4_1PctTo2X_I1/pp/atmos/ts/daily/tasmin_A2.02010101-02201231.nc" 
> > ncoords="7300" />
> >               </aggregation>
> >             </netcdf>
> >          </aggregation>
> >        </netcdf>
> >    </dataset>
> >
> >
> > I'm open to any suggestions or ideas!
> >
> > thanks -
> > kevin
> >
> >
> > John Caron wrote:
> > > Hi Kevin:
> > >
> > > I havent had time to reproduce this yet, but im guessing one source 
> > > of the slowdown is using opendap URLS in the compound aggregation. 
> > > It would be interesting to time 1) the single aggregations, 2) the 
> > > compound agg as it exists, and 3) the compound agg, but replace the 
> > > opendap URLs with direct netcdf files,
> > >
> > > see attached file

--
Kevin O'Brien                   UW/JISAO	
Research Scientist              NOAA/PMEL/TMAP
206-526-6751                    http://tmap.pmel.noaa.gov

"The contents of this message are mine personally and do 
not necessarily reflect any position of the Government 
or the  National Oceanic and Atmospheric Administration."