[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: data file compression / cache / bz2?

Hi Rob:

The files should be in content/thredds/cache/ and YDS allows up to 1 Gbyte before deleting oldest files. It is possible to change the place, but not yet the maximum size of the cache. Another possible cache stategy is to delete files after a certain age (which makes more sense for a rolling archive like motherlode).

Im going to make some/all of this user settable. what would you like to do?

If file ends with ".Z", ".zip", ".gzip", ".gz", or ".bz2", it will 
uncompress/unzip and write to new file without the suffix.

Rob Cermak wrote:

Is there a way to have THREDDS systematically clean up uncompressed
data files?   We noticed that THREDDS can support compressed (gzip)
NetCDF files, but the uncompressed files linger.

A periodic program can be run to scan through and remove them, but
this would assume that the periodic cleaning would keep up with the
rate of uncompression as data is accessed.

Maybe something similar to OPeNDAP Server 3 (dods_cache)?  Even this
has its limits.  If the cache size is not high enough, with enough
traffic, requests will get stomped.   Or just a table in memory
tracking uncompressed files.  If the server/system ever crashed,
then this would leave stuff about.

We were wondering if the same compression support can be extended to
include bzip2 compression (bz2)?


=============================================================================== To unsubscribe thredds, visit: http://www.unidata.ucar.edu/mailing-list-delete-form.html ===============================================================================

NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.