[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: phone message -- sample XML to follow (here it is)


Hi John -
I did as you suggested and moved the ncoords attribute to the outer aggregation and I was able to get to the aggregation in around 28 seconds. Just to confirm that it wasn't something system-related, I changed the xml configuration back, and verified that when the ncoords attribute was in the inner aggregation, it took around 2 minutes to open. So that's a big speed improvement! I think I will expand the aggregation xml to include full experiments and see how the performance changes. 

By the way, you can get all of these files at
ftp://nomads.gfdl.noaa.gov/gfdl_cm2_0/CM2Q-d2_1PctTo4x_j1/pp/atmos/ts/daily/
and you'll see there are many more that would actually be configured into the aggregation..
One thing I did notice and have a question about - after I moved the ncoords attribute to the outer aggregation, and I restarted the server - a cache file showed up in the cacheAged directory. When I then just restarted the server to test the use of cache, after I had opened the aggregation (which again took around 30 seconds), I noticed that the cache file in the cacheAged directory had apparently been updated (at least the time stamp of the file was new). If nothing in the aggregation has changed, should it be updating the cache file? Or should it use the cache file already there? 

thanks -
kevin

John Caron wrote:

Hi Kevin, Steve:

You should try putting the ncoords attribute on the outer aggregation:
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2"; ncoords="36500"> 
              <aggregation type="union">
<netcdf location="file:/data/gfdl_cm2_0/CM2Q-d2_1PctTo4x_j1/pp/atmos/ts/daily/pr_A2.00010101-01001231.nc" /> <netcdf location="file:/data/gfdl_cm2_0/CM2Q-d2_1PctTo4x_j1/pp/atmos/ts/daily/tasmax_A2.00010101-01001231.nc" /> <netcdf location="file:/data/gfdl_cm2_0/CM2Q-d2_1PctTo4x_j1/pp/atmos/ts/daily/tasmin_A2.00010101-01001231.nc" /> 
              </aggregation>
            </netcdf>

let me know if that helps.
Id like to test this nested aggregation as a use case . Can I get those 9 files? thanks. 


Steve Hankin wrote:
(This is a continuation of the conversation that Kevin O'Brien started with you.) 

Hi John,
Below is the ncML and TDS configuration information. It all "works" ... except the caching. Any clues? 

     - Steve

===

This from threddsConfig.xml

  <AggregationCache>
<dir>/home/pmel/DataPortal/apache-tomcat-5.5.25/content/thredds/cacheAged/</dir> 
    <scour>24 hours</scour>
    <maxAge>90 days</maxAge>
</AggregationCache> 
===
And this is the latest ncML that Kevin tested: "It took nearly two minutes to open the aggregation the first time. After that, accesses were quick -- evidently caching was working. Then I restarted the tomcat server, and again it took nearly two minutes to open the aggregation. I could see that the cache file in the caching directory was again updated after the second tomcat restart (ie, the cache was rewritten rather than used)..." 

<catalog name="test IPCC Datasets"
xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"; 
        xmlns:xlink="http://www.w3.org/1999/xlink";>
<service name="thisDODS3" serviceType="OpenDAP" base="/thredds/dodsC/" /> <dataset ID="CM2Q-d2_1PctTo4x_j1 atmos daily all vars 00010101-03001231 test" name="CM2Q-d2_1PctTo4x_j1 atmos daily all vars 00010101-03001231 test" urlPath="ipcc_ar4_CM2.0_R1_1to4x-0_daily_atmos_00010101-03001231_test"> 
        <serviceName>thisDODS3</serviceName>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";> 
          <aggregation dimName="time" type="joinExisting">
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";> 
               <aggregation type="union">
<netcdf location="file:/data/gfdl_cm2_0/CM2Q-d2_1PctTo4x_j1/pp/atmos/ts/daily/pr_A2.00010101-01001231.nc" ncoords="36500" /> <netcdf location="file:/data/gfdl_cm2_0/CM2Q-d2_1PctTo4x_j1/pp/atmos/ts/daily/tasmax_A2.00010101-01001231.nc" ncoords="36500" /> <netcdf location="file:/data/gfdl_cm2_0/CM2Q-d2_1PctTo4x_j1/pp/atmos/ts/daily/tasmin_A2.00010101-01001231.nc" ncoords="36500" /> 
               </aggregation>
             </netcdf>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";> 
               <aggregation type="union">
<netcdf location="file:/data/gfdl_cm2_0/CM2Q-d2_1PctTo4x_j1/pp/atmos/ts/daily/pr_A2.01010101-02001231.nc" ncoords="36500" /> <netcdf location="file:/data/gfdl_cm2_0/CM2Q-d2_1PctTo4x_j1/pp/atmos/ts/daily/tasmax_A2.01010101-02001231.nc" ncoords="36500" /> <netcdf location="file:/data/gfdl_cm2_0/CM2Q-d2_1PctTo4x_j1/pp/atmos/ts/daily/tasmin_A2.01010101-02001231.nc" ncoords="36500" /> 
               </aggregation>
             </netcdf>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";> 
               <aggregation type="union">
<netcdf location="file:/data/gfdl_cm2_0/CM2Q-d2_1PctTo4x_j1/pp/atmos/ts/daily/pr_A2.02010101-03001231.nc" ncoords="36500"/> <netcdf location="file:/data/gfdl_cm2_0/CM2Q-d2_1PctTo4x_j1/pp/atmos/ts/daily/tasmax_A2.02010101-03001231.nc" ncoords="36500"/> <netcdf location="file:/data/gfdl_cm2_0/CM2Q-d2_1PctTo4x_j1/pp/atmos/ts/daily/tasmin_A2.02010101-03001231.nc" ncoords="36500" /> 
               </aggregation>
             </netcdf>
          </aggregation>
        </netcdf>
      </dataset>

</catalog>




Steve Hankin wrote:

Hi John,
We're at the phone number in the signature line below. Will follow this email shortly with some XML fragments ... hoping maybe you have a suggestion. 

   - Steve



--
Steve Hankin, NOAA/PMEL -- address@hidden
7600 Sand Point Way NE, Seattle, WA 98115-0070
ph. (206) 526-6080, FAX (206) 526-6744

"The only thing necessary for the triumph of evil is for good men
to do nothing." -- Edmund Burke



--
Kevin O'Brien                   UW/JISAO        
Research Scientist              NOAA/PMEL/TMAP
206-526-6751                    http://www.pmel.noaa.gov
"The contents of this message are mine personally and do not necessarily reflect any position of the Government or the National Oceanic and Atmospheric Administration."