Re: [thredds] asynchronous data access

John Cartwright wrote:
Hello All,

has anyone used TDS for asynchronous data access, e.g. data in a tape
library that might take minutes to an hour to retrieve?

Hi John,
After my RAMADDA talk last week I spoke with someone from NGDC about just that and that has gotten me thinking about this issue.

I have actually implemented something like this in RAMADDA for remote files in remote FTP sites:

For the FTP service the user specifies a size threshold (e.g., 10MB). When a service in RAMADDA (e.g., opendap) requires a file the FTP service looks in its local cache. If not there and if its below the size threshold then the file is copied over from the FTP server and stored in the cache. One of the problems with going to a tape store is the latency in fetching the file. But, that is only suffered the first time or when the file is flushed out of the cache.

I've also lately been thinking about integrating with cloud storage (e.g., amazon s3). This would work just like the FTP service. You have a local cache and the smarts to fetch it from the cloud storage. Another approach to accessing cloud storage would be to do it at the file system level not at the application level, e.g. FUSE:


  • 2010 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: