[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [thredds] announce stable release 4.3.19



Hi Joe,

The datasetScan catalog generation uses the thredds.crawlabledataset
package behind the scenes. (It is in the cdm part of the code.)

The CrawlableDataset (CrDs) interface has a CrawlableDatsetFile
implementation that handles crawling file systems. There is also a very
old CrawlableDatasetDods implementation that tried to crawl and scrape
the HTML interface on the old DODS servers. Needless to say, it was
pretty brittle. We never pushed its use.

I think that writing a CrawlableDataset implementation that reads and
crawls THREDDS XML catalogs would probably accomplish what you are
trying to do. Alternate CrDs impls can be specified in a datasetScan
element. I'll have to remember how to do that. We've thought about this
at times in the past, just never got to actually working on it.

Anyway, one thought on how to proceed.
Have a good weekend,

Ethan


On 10/18/2013 11:18 AM, Joe Lee wrote:
> Hi, John!
> 
> Thanks for the tip of putting NcML inside the catalog.xml! It looks
> more handy than writing hundreds of individual NcML file and put them
> under dataRoot.
> 
> I'm just wondering if there's a solution to automate this dynamically
> without writing an explicit NcML style catalog generation  tool  for
> individual data under Hyrax.  I'm talking about a new THREDDS
> capability that can crawl all  data available served by Hyrax
> server(s) and make them ready (as if they are local file on THREDDS
> sever) to serve all  fancy services (WMS, ncISO, netCDF subset) that
> are only available through THREDDS.
> 
> For example,  I wish THREDDS can handle this kind rule inside
> catalog.xml:
> 
> <datasetScan name="Remote Hyrax server", ID="testHyraxDataScan"
> path="hyrax" location="http://hyrax.top.level/opendap/data/";> 
> <filter><include wildcard="*.hdf"/></filter> <!-- serve only remote
> HDF4 data, not HDF5 --> </datasetScan>
> 
> Do you think it's a good idea to pursue?
> 
>  Regards, 
>  
> 
> -----Original Message-----
> From: John Caron [mailto:address@hidden] 
> Sent: Friday, October 18, 2013 11:46 AM
> To: Joe Lee
> Cc: Ethan Davis; Ted Habermann; address@hidden
> Subject: Re: [thredds] announce stable release 4.3.19
> 
> Hi Joe:
> 
> On our Test server, I have:
> 
> <?xml version="1.0" encoding="UTF-8"?>
> <catalog name="TDS Test Server"
>          
> xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0";
>          xmlns:xlink="http://www.w3.org/1999/xlink";>
> 
>   <service name="all" base="" serviceType="compound">
>     <service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/"/>
>     <service name="wcs" serviceType="WCS" base="/thredds/wcs/"/>
>     <service name="wms" serviceType="WMS" base="/thredds/wms/"/>
>     <service name="ncss" serviceType="NetcdfSubset" base="/thredds/ncss/"/>
>     <service name="cdmremote" serviceType="CdmRemote" 
> base="/thredds/cdmremote/"/>
>     <service name="iso" serviceType="ISO" base="/thredds/iso/"/>
>     <service name="ncml" serviceType="NCML" base="/thredds/ncml/"/>
>     <service name="uddc" serviceType="UDDC" base="/thredds/uddc/"/>   
>   </service>
> 
> 
>   <dataset name="Test Hyrax" ID="Hyrax2TDS" serviceName="all" 
> urlPath="Hyrax2TDS/AIRS.2002.08.01.L3.RetStd_H031.v4.0.21.0.G06104133732.hdf" 
> dataType="Grid">
>     <ncml:netcdf 
> xmlns:ncml="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2"; 
>         
> location="http://eosdap.hdfgroup.uiuc.edu:8080/opendap/data/NASAFILES/hdf4/AIRS.2002.08.01.L3.RetStd_H031.v4.0.21.0.G06104133732.hdf";
>  />
>   </dataset>
> 
> </catalog>
> 
> which exposes the Hyrax dataset through all the TDS services.
> The urlPath is arbitrary but it seems like just echoing the hyrax file name 
> is a good thing to do.
> you probably would want to generate these catalogs with a script. also may 
> have trouble scaling up if you had thousands of them, since the ncml gets 
> cached in memory.
> 
> Seems to work ok. It would be good to test this thoroughly. Do you have a TDS 
> running?
> 
> John
> 
> 
> 
> On 10/17/2013 1:49 PM, Joe Lee wrote:
>> Thanks, John!
>> FYI, I succeeded in building TDS-4.3.19 with maven.
>>
>>    By the way, I have one question.
>>
>>    Do you think it's possible to serve remote data served by Hyrax OPeNDAP 
>> server through THREDDS?
>>
>>    I know I can write a simple NcML file as a wrapper and put it under 
>> THREDDS public/ data directory but I'd like to serve OPeNDAP data directly 
>> via URL.
>> Since netCDF-Java can access remote OPeNDAP resource via nc_open() call 
>> underneath, I think I can simply replace the local file name with URL in 
>> theory.
>> I'm wondering if anyone has done a similar work of accessing OPeNDAP 
>> directly with THREDDS.
>>
>>    The purpose is to allow access NASA data served by Hyrax via THREDDS so 
>> that users can utilize the THREDDS'  WMS/ncISO/NetCDF subset  capability. So 
>> far, many NASA HDF data products  can be visualized by netCDF-Java client 
>> like IDV/Panoply already because they follow the CF-conventions through our 
>> Hyrax HDF handler work.  My next goal is to integrate THREDDS with Hyrax to 
>> provide additional services.
>>
>>    Any input or suggestion is welcome!
>>
>> -----Original Message-----
>> From: John Caron [mailto:address@hidden]
>> Sent: Thursday, October 17, 2013 2:24 PM
>> To: Joe Lee
>> Cc: Ethan Davis; Ted Habermann
>> Subject: Re: [thredds] announce stable release 4.3.19
>>
>> the gradle stuff is not ready to use yet - expect it will require IntelliJ 
>> 13. WE are still using maven, so you should "import maven project".
>>
>> On 10/17/2013 1:11 PM, H. Joe Lee wrote:
>>> Hi, John!
>>>
>>>     I tried to build the new Thredds but have an issue with 
>>> IntelliJ/Gradle on my Windows 7.
>>>
>>>     Would you please take a look at my screenshot and tell me what could be 
>>> wrong?
>>>
>>>     I git pulled the source from the documentation link that you sent.
>>>
>>> --
>>> HDF: Software that Powers Science
>>>
>>>
>>> On Tue, Oct 8, 2013 at 5:26 PM, John Caron <address@hidden> wrote:
>>>> The Netcdf-Java library and TDS stable release 4.3.19 is now ready:
>>>>
>>>>     
>>>> http://www.unidata.ucar.edu/software/thredds/current/netcdf-java/documentation.htm
>>>>     
>>>> http://www.unidata.ucar.edu/software/thredds/current/tds/TDS.html
>>>>
>>>> (note that the above URLs are now the standard way to get the stable 
>>>> releases. Please update your bookmarks)
>>>>
>>>> This is entirely a (minor) bug fix.
>>>>
>>>>
>>>> CDM
>>>>    * error assigning gregorian calendar as default to CF files
>>>>    * grib: protect against null vertical level name
>>>>    * grib time partition : ignore groups not in canonical partition
>>>>    * grib: fnmoc tables, try to improve
>>>>    * hdf5: deep error in FractalHeap
>>>>    * ncml: was ignoring separators in <values> tag
>>>>
>>>> TDS
>>>>    * ncss fixes
>>>>    * updated ncWms version (20130918)
>>>>
>>>>
>>>> thanks!
>>>>
>>>> _______________________________________________
>>>> thredds mailing list
>>>> address@hidden
>>>> For list information or to unsubscribe,  visit:
>>>> http://www.unidata.ucar.edu/mailing_lists/

-- 
Ethan Davis                                       UCAR Unidata Program
address@hidden                    http://www.unidata.ucar.edu