Re: [thredds] THREDDS Data Server serving from Amazon S3

        Hi David-

        Does THREDDS require a mounted block filesystem to operate or can it 
reference HTTP resources?  S3 is HTTP-based access to objects 
(files/pictures/blobs).  If a mounted filesystem is required, then you have to 
mount an instance of EBS (Elastic Block Store), which is their virtual 
filesystem, to your EC2 instance.  I haven't done this, but it looks like it's 
common practice to construct disk images as snapshots and store those images to 
S3.  They are not meant to be mutable while on S3, but you can import an S3 
image back to EBS and mount it, most especially in a replicated way to many 
servers.

        Just wanted to point out the distinction.

        -Rob


> On Jul 13, 2015, at 6:59 PM, David Nahodil <David.Nahodil@xxxxxxxxxxx> wrote:
> 
> Hi all,
> 
> We are looking at moving our THREDDS Data Server to Amazon EC2 instances with 
> the data hosted on S3. I'm just wondering if anyone has tried using TDS with 
> data hosted on S3?
> 
> I had a quick back-and-forth with Sean at Unidata (see below) about this.
> 
> Regards,
> 
> David
> 
> > > Unfortunately, I do not know of anyone who has done this, although we 
> > > have had at lease one other person ask. From what I understand, there is 
> > > a way to mount an S3 storage as a virtual file system, in which case I 
> > > would *think* that the TDS would work as it normally does (depending on 
> > > the kind of data you have).
> 
> > We have considered mounting the S3 storage as a filesystem and running it 
> > like that. However, our feeling was that the tools were not really 
> > production ready and that we're really misrepresenting S3 by pretending it 
> > is a file system. So this is why we're investigating if anyone has used TDS 
> > with the S3 API directly.
> 
> > > What kind of data do you have? Will your TDS also be in the cloud? Do you 
> > > plan on serving the data inside of amazon to other EC2 instances, or do 
> > > you plan on crossing the cloud/commodity web boundary with the data, in 
> > > which case that could get very expensive quite quickly?
> 
> > We have about 2 terabytes of marine and climate data that we are currently 
> > serving from our existing infrastructure. The plan is to move the 
> > infrastructure to Amazon Web Services so TDS would be hosted on EC2 
> > machines and the data on S3. We're hoping this setup should work okay, but 
> > we might still have a hurdle or two to come. :)
> 
> > We have someone here who once wrote a plugin/adapter for TDS to work with 
> > an obscure filesystem that our data used to be stored on. So we have a 
> > little experience in what might be involved in what might be involved for 
> > doing the same with S3. We just wanted to make sure that if anyone had done 
> > some work already that we made use of that.
> 
> > > We very, very recently (as in a day ago) got some Amazon resources to 
> > > play around on, but we won't have a chance to kick those tires until 
> > > after our training workshops at the end of the month.
> 
> 
> University of Tasmania Electronic Communications Policy (December, 2014). 
> This email is confidential, and is for the intended recipient only. Access, 
> disclosure, copying, distribution, or reliance on any of it by anyone outside 
> the intended recipient organisation is prohibited and may be a criminal 
> offence. Please delete if obtained in error and email confirmation to the 
> sender. The views expressed in this email are not necessarily the views of 
> the University of Tasmania, unless clearly intended otherwise.
> _______________________________________________
> thredds mailing list
> thredds@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe,  visit: 
> http://www.unidata.ucar.edu/mailing_lists/

  • 2015 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: