Re: [bww-users] Community-managed cloud storage services

  • To: Carlos Maltzahn <carlosm@xxxxxxxxxxxx>
  • Subject: Re: [bww-users] Community-managed cloud storage services
  • From: Mohan Ramamurthy <mohan@xxxxxxxx>
  • Date: Sun, 14 Feb 2016 06:52:14 -0700
It is true that OCC is operating its own infrastructure and not leveraging commercial cloud infrastructure. Not sure what you mean by community-managed, but the projects that are hosted and the decisions on services, datasets, etc. are based on discussions with members and community participants.

Mohan

On 2/13/16 9:09 PM, Carlos Maltzahn wrote:
Reading the About page <http://occ-data.org/about/>:

To better understand our role, it is helpful to divide projects in these areas into three groups:

 1. Individual researchers and small projects typically do not need
    much computing infrastructure and can either operate their own or
    use a public cloud service provider such as Amazon.
 2. The OCC is designed to serve medium to large size research
    projects by managing and operating a cloud computing
    infrastructure that can be shared across these projects.
 3. Very large research projects, such as the LHC, the LSST, and the
    OOI, typically develop their own dedicated computing infrastructure.

It sounds like OCC is building their own cloud infrastructure instead of leveraging commercial cloud providers. Also, the membership benefits <http://occ-data.org/images/occ-fees-2016.pdf> do not include actual usage of the infrastructure and is more about participating in a standardization effort.

I’m interested in models of "community-managed” cloud storage services where the management involves cost and usage (as opposed to operating the hardware infrastructure). But I couldn’t find anything on the OCC web site that addresses that.

Or do I miss something?

Carlos

On Feb 11, 2016, at 2:49 PM, Mohan Ramamurthy <mohan@xxxxxxxx <mailto:mohan@xxxxxxxx>> wrote:

On 2/11/16 3:44 PM, Scott Collis wrote:
So is this along the same lines as AWS S3?

Yes.
Does it still rely on a download and compute framework?


At the moment, this is true but we are working to working to develop data-proximate, server-side processing/analysis capabilities by moving our wares (and client tools) to the cloud, and through the development and implementation of DAP4 protocol that supports asynchronous computing capabilities.

Mohan
Mohan Ramamurthy <mailto:mohan@xxxxxxxx>
February 11, 2016 at 4:42 PM
Carlos,

Unidata is working with Open Commons Consortium (http://occ-data.org/), which provides "community-managed" cloud storage and computing services. At the moment, Unidata's collaboration with OCC is focused on the NOAA Big Data project, but we expect that to grow beyond the scope of that project.

Mohan

On 2/11/16 2:34 PM, Carlos Maltzahn wrote:

_______________________________________________
bww-users mailing list
bww-users@xxxxxxxxxxxxxxxx
For list information, to unsubscribe, or change your membership options, visit:http://www.unidata.ucar.edu/mailing_lists/
Carlos Maltzahn <mailto:carlosm@xxxxxxxx>
February 11, 2016 at 4:34 PM
All,

This is a request for examples of community-managed cloud storage services where

  * “community-managed” means that the cost of the cloud storage
    service as well as its usage is managed by an institution
    serving a (scientific community), including very large
    communities such as earth sciences or smaller ones such as
    numerical weather prediction, and

  * “cloud storage services” are commercial, highly available
    “pay-as-you-go” services that provide safe and economic storage
    of large amounts of data and allow global sharing of that data
    controlled by the party who pays, but disappear as soon as
    payment for these services stop.


Today commercial cloud storage services are readily available and successfully hide the many technical challenges of highly available long-term storage at very attractive cost. Cloud storage also provides an excellent platform for naming and sharing large (and small) datasets which is essential for collaboration and reproducibility in data-intensive scientific disciplines. Yet science communities are slow to adopt cloud storage. There are probably many reasons for that but one that I repeatedly came across: the data stored in cloud storage disappears when funding for the service runs out.

If the availability of a particular data set depends on a single community member's availability of funding, the likelihood of loosing data can be quite high and makes cloud storage too brittle for a reliable medium for scientific data. A better approach might be to make the availability of all data sets depend on the availability of funding within an entire community. Such an arrangement would benefit that community by facilitating data sharing, collaboration, and maintaining greater reproducibility of scientific results.

But community-funded cloud storage has all the management challenges of a commons. For example, how should the storage space be governed? How much money should the community spend on cloud storage? How is the money raised among the members of the community? How do communities prevent The Tragedy of the Commons <https://en.wikipedia.org/wiki/Tragedy_of_the_commons>?

Please let me know of any examples you are aware of. Who is working on this? Do examples exist with somewhat different definitions of "community-managed" or "cloud storage services”?

Thanks,
Carlos


--
Carlos Maltzahn
Adjunct Professor
Computer Science Department
University of California, Santa Cruz
http://users.soe.ucsc.edu/~carlosm/

_______________________________________________
bww-users mailing list
bww-users@xxxxxxxxxxxxxxxx
For list information, to unsubscribe, or change your membership options, visit:http://www.unidata.ucar.edu/mailing_lists/


_______________________________________________
bww-users mailing list
bww-users@xxxxxxxxxxxxxxxx <mailto:bww-users@xxxxxxxxxxxxxxxx>
For list information, to unsubscribe, or change your membership options, visit:http://www.unidata.ucar.edu/mailing_lists/


  • 2016 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the bww-users archives: