Greetings David,
Sorry for the delay. Zarr is on the list for netCDF-Java to support
(development priority given to reading), as well as support for the
yet-to-be-released ncZarr. Zarr will be supported for traditional file
systems as well as object stores like S3.
Cheers,
Sean
On Wed, Mar 4, 2020 at 11:56 AM David Neufeld via thredds
<thredds@xxxxxxxxxxxxxxxx> wrote:
>
> Hi Ethan,
>
> That's great news!
>
> By way of follow-up can you share any forward looking plans the team may have
> related to zarr support in the future? Is this part of a roadmap for thredds,
> or is the expectation that netcdf developers can leverage the s3 support and
> then write to zarr as part of their own workflow?
>
> Thanks,
> Dave
>
>
> On Wed, Mar 4, 2020 at 9:53 AM Ethan Davis <edavis@xxxxxxxx> wrote:
>>
>> Hi Joe,
>>
>> [Sorry for the delayed response.]
>>
>> The S3 work moved to the Unidata/netCDF-java repo in PR #173 ("S3 Support").
>> This PR got merged into master a week or so ago and is available in the
>> netCDF-Java 5.3.0-SNAPSHOT release (and will be in the upcoming 5.3.0
>> release). The latest TDS code built with netCDF-Java 5.3.0-SNAPSHOT can be
>> configured to serve an individual netCDF file stored as an S3 object using a
>> datasetRoot configuration, e.g.
>>
>>
>> <?xml version="1.0" encoding="UTF-8"?>
>>
>> <catalog name="Test TDS S3"
>>
>> xmlns="https://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0";
>>
>> xmlns:xlink="https://www.w3.org/1999/xlink";
>>
>> xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance";
>>
>>
>> xsi:schemaLocation="https://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0
>>
>> https://www.unidata.ucar.edu/schemas/thredds/InvCatalog.1.0.6.xsd";>
>>
>>
>> <datasetRoot path="s3-test" location="s3://noaa-goes16" />
>>
>>
>> <dataset name="Test GOES-16 S3" ID="testS3Grid"
>>
>>
>> urlPath="s3-test/ABI-L1b-RadC/2019/363/21/OR_ABI-L1b-RadC-M6C16_G16_s20193632101189_e20193632103574_c20193632104070.nc"
>>
>> dataType="Grid"/>
>>
>>
>> </catalog>
>>
>>
>> In this case, the datasetRoot location is the bucket name, and the urlPath
>> is the datasetRoot path combined with the key. We rely on the AWS Java SDK
>> (v2) to handle credentials, setting of region, etc. For now, you can set the
>> region by creating a credentials file ~/.aws/credentials that looked like:
>>
>>
>> [default]
>>
>> region=us-east-1
>>
>>
>> Which is how netCDF-java knows which region to use for bucket access. We may
>> look at other mechanisms to make that a bit more integrated into TDS
>> configuration but for now that should work.
>>
>>
>> Once the netCDF 5.3.0 release comes out, TDS snapshot builds will be built
>> with this capability. For now, you would need to build the TDS and
>> explicitly tell it to build with netCDF-Java 5.3.0-SNAPSHOT.
>>
>> Cheers,
>>
>> Ethan
>>
>> On Tue, Feb 4, 2020 at 2:30 PM H. Joe Lee <hyoklee@xxxxxxxxxxxx> wrote:
>>>
>>> Hi,
>>>
>>> Is it possible to serve netCDF data on AWS S3 using THREDDS?
>>> I think it seems possible based on the S3 feature branch [1].
>>>
>>> If so, can someone share an example THREDDS catalog configuration?
>>>
>>> Regards,
>>>
>>> [1] https://github.com/Unidata/thredds/tree/feature/s3+hdfs
>>>
>>>
>>> _______________________________________________
>>> NOTE: All exchanges posted to Unidata maintained email lists are
>>> recorded in the Unidata inquiry tracking system and made publicly
>>> available through the web. Users who post to any of the lists we
>>> maintain are reminded to remove any personal information that they
>>> do not want to be made public.
>>>
>>>
>>> thredds mailing list
>>> thredds@xxxxxxxxxxxxxxxx
>>> For list information or to unsubscribe, visit:
>>> https://www.unidata.ucar.edu/mailing_lists/
>>
>> _______________________________________________
>> NOTE: All exchanges posted to Unidata maintained email lists are
>> recorded in the Unidata inquiry tracking system and made publicly
>> available through the web. Users who post to any of the lists we
>> maintain are reminded to remove any personal information that they
>> do not want to be made public.
>>
>>
>> thredds mailing list
>> thredds@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> https://www.unidata.ucar.edu/mailing_lists/
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> thredds mailing list
> thredds@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> https://www.unidata.ucar.edu/mailing_lists/
