Re: [thredds] Catalog example for AWS S3 resource?

  • To: Ethan Davis <edavis@xxxxxxxx>
  • Subject: Re: [thredds] Catalog example for AWS S3 resource?
  • From: "H. Joe Lee" <hyoklee@xxxxxxxxxxxx>
  • Date: Mon, 9 Mar 2020 09:39:15 -0500
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=hdfgroup.org; dmarc=pass action=none header.from=hdfgroup.org; dkim=pass header.d=hdfgroup.org; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=PvsHRbdKli9flvqGF+h04vnQq+zozk8hsW+bW6ZH2FY=; b=nz4M5jId0IjMkBd/HGCFaH/bVPxbb/NgRdRrFOxqCYDS65WcBWht4v5ohMgZC8l2p0UYtNURWXmfLO/RYyqIb7QMA00/J9jTLlfOaYKEdgUOSlV+Txo4W68RlDrSbdd0EqZmjIVmHW1DtdpwxltWt+wJPaY3UT8+/nxa4VHvyOjY7iqTykc8ZMWwmmrOcWrub3960cOQdn0yS+vFBk1XixxYQ9lcjQFCmmSikejfSBg+bl0QcbhMklPQRmxneEdc5nPtssAIvog4fBIlMYauN4iNZs0xY4HyyVHJvrp9jcSHOmGhFsplfbh7/zhmkB7jkrcHOkY5/dI5l7kq7VWSCA==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nq8aiM3CK5ao3h5smxjENWxXPyRAQ+a/mqS133ZqANWjbal+4+TMIA+zHHbtDK9/cyqDP0t8H8Gl4x7CeatJczMCmS4PrOq6ZZXEFbME7a9u0G9mBHzT0Q+MqAOlG0A1vXBO8vE2jeuZN4aZRMsYekV5NBII1T/m1WERbYVljX7DQEe2NhsGKdwc3wUhLdQ9wvmgrax5JoG2vf+TUv4wW2F//kAK8qfH5d6B5tAVqdvPkOW0CidIHh5eQX1jzCpY9UBUCz1chMytBgtxUKjGGnaggqCzyxludyWUdxIJ1f74y10LXuOLaadiLCu3IDXG4RHy8zPS/iOBIffIzbKWPg==
  • Authentication-results: spf=none (sender IP is ) smtp.mailfrom=hyoklee@xxxxxxxxxxxx;
  Thanks, Ethan!

  It's so cool to see toolsUI can access NASA HDF-EOS5 on S3.
I hope both IDV and Panoply can use the new netCDF-Java soon, too.

  By the way, will the master branch of THREDDS use the latest netCDF-java?
If not, where should I modify in the THREDDS source code to build
 THREDDS with netCDF-Java snapshot?

  I'm very excited to try the new THREDDS catalog with S3 datasetRoot path!

Sincerely,

--
Datafy everything in HDF for faster AI.




On Wed, Mar 4, 2020 at 10:52 AM Ethan Davis <edavis@xxxxxxxx> wrote:

> Hi Joe,
>
> [Sorry for the delayed response.]
>
> The S3 work moved to the Unidata/netCDF-java repo in PR #173
> <https://github.com/Unidata/netcdf-java/pull/173> ("S3 Support"). This PR
> got merged into master a week or so ago and is available in the netCDF-Java
> 5.3.0-SNAPSHOT release (and will be in the upcoming 5.3.0 release). The
> latest TDS code built with netCDF-Java 5.3.0-SNAPSHOT can be configured to
> serve an individual netCDF file stored as an S3 object using a datasetRoot
> configuration, e.g.
>
> <?xml version="1.0" encoding="UTF-8"?>
>
> <catalog name="Test TDS S3"
>
>   xmlns="https://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0";
>
>   xmlns:xlink="https://www.w3.org/1999/xlink";
>
>   xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance";
>
>   xsi:schemaLocation="
> https://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0
>
>     https://www.unidata.ucar.edu/schemas/thredds/InvCatalog.1.0.6.xsd";>
>
>   <datasetRoot path="s3-test" location="s3://noaa-goes16" />
>
>   <dataset name="Test GOES-16 S3" ID="testS3Grid"
>
>
>      
> urlPath="s3-test/ABI-L1b-RadC/2019/363/21/OR_ABI-L1b-RadC-M6C16_G16_s20193632101189_e20193632103574_c20193632104070.nc"
>
>            dataType="Grid"/>
>
> </catalog>
>
> In this case, the datasetRoot location is the bucket name, and the urlPath
> is the datasetRoot path combined with the key. We rely on the AWS Java SDK
> (v2) to handle credentials, setting of region, etc. For now, you can set
> the region by creating a credentials file ~/.aws/credentials that looked
> like:
>
> [default]
>
> region=us-east-1
>
> Which is how netCDF-java knows which region to use for bucket access. We
> may look at other mechanisms to make that a bit more integrated into TDS
> configuration but for now that should work.
>
> Once the netCDF 5.3.0 release comes out, TDS snapshot builds will be built
> with this capability. For now, you would need to build the TDS and
> explicitly tell it to build with netCDF-Java 5.3.0-SNAPSHOT.
>
> Cheers,
>
> Ethan
>
> On Tue, Feb 4, 2020 at 2:30 PM H. Joe Lee <hyoklee@xxxxxxxxxxxx> wrote:
>
>> Hi,
>>
>>   Is it possible to serve netCDF data on AWS S3 using THREDDS?
>>   I think it seems possible based on the S3 feature branch [1].
>>
>>   If so, can someone share an example THREDDS catalog configuration?
>>
>>   Regards,
>>
>> [1] https://github.com/Unidata/thredds/tree/feature/s3+hdfs
>>
>>
>> _______________________________________________
>> NOTE: All exchanges posted to Unidata maintained email lists are
>> recorded in the Unidata inquiry tracking system and made publicly
>> available through the web.  Users who post to any of the lists we
>> maintain are reminded to remove any personal information that they
>> do not want to be made public.
>>
>>
>> thredds mailing list
>> thredds@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe,  visit:
>> https://www.unidata.ucar.edu/mailing_lists/
>>
>
  • 2020 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: