Re: [thredds] How to control caching of Aggregated datasets?

• To: thredds@xxxxxxxxxxxxxxxx
• Subject: Re: [thredds] How to control caching of Aggregated datasets?
• From: John Caron <caron@xxxxxxxxxxxxxxxx>
• Date: Fri, 20 May 2011 15:00:31 -0600

Hi Heiko:

Ive added this sentence:

The recheckEvery attribute only applies when using a scan element.

to the various documents you've suggested. Thanks for pointing this out.

John

On 3/11/2011 1:31 AM, Heiko Klein wrote:

Thanks Mike,

that explains why I couldn't get it working. We could get your 'dir with links and scan' solution working for us, too.

Since the scripts creating the aggregation are written by some colleges, I need to convince them. It would help me if I could get a copy of your discussion with John?

And if it's going to be a long term issue, it might be nice to see a warning in http://www.unidata.ucar.edu/projects/THREDDS/tech/tds4.2/tutorial/NcML.htm and http://www.unidata.ucar.edu/software/netcdf/ncml/v2.2/Aggregation.html

Best regards,

Heiko

On 2011-03-10 18:05, McCann, Mike wrote:

Hello Heiko,

I was experiencing the same issue and after some discussion with John Caron

I switched my catalog to use a scan aggregation:

<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
<aggregation dimName="TIME" type="joinExisting" recheckEvery="2 hour">
<scan location="<dir>" suffix="_R_TS.nc" subdirs="false" />
</aggregation>
</netcdf>

I needed to create a<dir>  with symbolic links to all the files being

aggregated. The last file in the aggregation is recreated every two hours

and so far this technique has been working fine.

Hope this helps,
Mike


On 3/10/11 8:44 AM, "Heiko Klein"<Heiko.Klein@xxxxxx>  wrote:

Hi,

I have a thredds aggregated dataset, with the setup in catalog.xml:
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
<aggregation dimName="time" type="joinExisting" recheckEvery="10
seconds">
<netcdf location="/file1.nc" coordValue="1" />
<netcdf location="/file2.nc" coordValue="2" />
</aggregation>
</netcdf>


After changing the aggregation, i.e. by adding a new file, e.g.
file3.nc, or by changing the coordValue, I usually run reinitialize the
catalogues with /thredds/admin/debug?catalogs/reinit

The changes appear in the opendap stream first after a while (some
hours?). The only way I currently found to make the correct results

really appear in opendap is to do the following shortly after each others:

a) clear file object caches
b) delete the corresponding aggregation cache file from the disk.
(And I'm not sure if this allways works?)

Is there a better way to control aggregation changes in the catalog.xml

file, i.e. to flush any aggregation-caches when the aggregation changes?

Best regards,

Heiko

_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  visit:
http://www.unidata.ucar.edu/mailing_lists/

• References:
  • Re: [thredds] How to control caching of Aggregated datasets?
    • From: McCann, Mike
  • Re: [thredds] How to control caching of Aggregated datasets?
    • From: Heiko Klein