Hi Joe, The datasetScan catalog generation uses the thredds.crawlabledataset package behind the scenes. (It is in the cdm part of the code.) The CrawlableDataset (CrDs) interface has a CrawlableDatsetFile implementation that handles crawling file systems. There is also a very old CrawlableDatasetDods implementation that tried to crawl and scrape the HTML interface on the old DODS servers. Needless to say, it was pretty brittle. We never pushed its use. I think that writing a CrawlableDataset implementation that reads and crawls THREDDS XML catalogs would probably accomplish what you are trying to do. Alternate CrDs impls can be specified in a datasetScan element. I'll have to remember how to do that. We've thought about this at times in the past, just never got to actually working on it. Anyway, one thought on how to proceed. Have a good weekend, Ethan On 10/18/2013 11:18 AM, Joe Lee wrote: > Hi, John! > > Thanks for the tip of putting NcML inside the catalog.xml! It looks > more handy than writing hundreds of individual NcML file and put them > under dataRoot. > > I'm just wondering if there's a solution to automate this dynamically > without writing an explicit NcML style catalog generation tool for > individual data under Hyrax. I'm talking about a new THREDDS > capability that can crawl all data available served by Hyrax > server(s) and make them ready (as if they are local file on THREDDS > sever) to serve all fancy services (WMS, ncISO, netCDF subset) that > are only available through THREDDS. > > For example, I wish THREDDS can handle this kind rule inside > catalog.xml: > > <datasetScan name="Remote Hyrax server", ID="testHyraxDataScan" > path="hyrax" location="http://hyrax.top.level/opendap/data/"> > <filter><include wildcard="*.hdf"/></filter> <!-- serve only remote > HDF4 data, not HDF5 --> </datasetScan> > > Do you think it's a good idea to pursue? > > Regards, > > > -----Original Message----- > From: John Caron [mailto:address@hidden > Sent: Friday, October 18, 2013 11:46 AM > To: Joe Lee > Cc: Ethan Davis; Ted Habermann; address@hidden > Subject: Re: [thredds] announce stable release 4.3.19 > > Hi Joe: > > On our Test server, I have: > > <?xml version="1.0" encoding="UTF-8"?> > <catalog name="TDS Test Server" > > xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0" > xmlns:xlink="http://www.w3.org/1999/xlink"> > > <service name="all" base="" serviceType="compound"> > <service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/"/> > <service name="wcs" serviceType="WCS" base="/thredds/wcs/"/> > <service name="wms" serviceType="WMS" base="/thredds/wms/"/> > <service name="ncss" serviceType="NetcdfSubset" base="/thredds/ncss/"/> > <service name="cdmremote" serviceType="CdmRemote" > base="/thredds/cdmremote/"/> > <service name="iso" serviceType="ISO" base="/thredds/iso/"/> > <service name="ncml" serviceType="NCML" base="/thredds/ncml/"/> > <service name="uddc" serviceType="UDDC" base="/thredds/uddc/"/> > </service> > > > <dataset name="Test Hyrax" ID="Hyrax2TDS" serviceName="all" > urlPath="Hyrax2TDS/AIRS.2002.08.01.L3.RetStd_H031.v18.104.22.168.G06104133732.hdf" > dataType="Grid"> > <ncml:netcdf > xmlns:ncml="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2" > > location="http://eosdap.hdfgroup.uiuc.edu:8080/opendap/data/NASAFILES/hdf4/AIRS.2002.08.01.L3.RetStd_H031.v22.214.171.124.G06104133732.hdf" > /> > </dataset> > > </catalog> > > which exposes the Hyrax dataset through all the TDS services. > The urlPath is arbitrary but it seems like just echoing the hyrax file name > is a good thing to do. > you probably would want to generate these catalogs with a script. also may > have trouble scaling up if you had thousands of them, since the ncml gets > cached in memory. > > Seems to work ok. It would be good to test this thoroughly. Do you have a TDS > running? > > John > > > > On 10/17/2013 1:49 PM, Joe Lee wrote: >> Thanks, John! >> FYI, I succeeded in building TDS-4.3.19 with maven. >> >> By the way, I have one question. >> >> Do you think it's possible to serve remote data served by Hyrax OPeNDAP >> server through THREDDS? >> >> I know I can write a simple NcML file as a wrapper and put it under >> THREDDS public/ data directory but I'd like to serve OPeNDAP data directly >> via URL. >> Since netCDF-Java can access remote OPeNDAP resource via nc_open() call >> underneath, I think I can simply replace the local file name with URL in >> theory. >> I'm wondering if anyone has done a similar work of accessing OPeNDAP >> directly with THREDDS. >> >> The purpose is to allow access NASA data served by Hyrax via THREDDS so >> that users can utilize the THREDDS' WMS/ncISO/NetCDF subset capability. So >> far, many NASA HDF data products can be visualized by netCDF-Java client >> like IDV/Panoply already because they follow the CF-conventions through our >> Hyrax HDF handler work. My next goal is to integrate THREDDS with Hyrax to >> provide additional services. >> >> Any input or suggestion is welcome! >> >> -----Original Message----- >> From: John Caron [mailto:address@hidden >> Sent: Thursday, October 17, 2013 2:24 PM >> To: Joe Lee >> Cc: Ethan Davis; Ted Habermann >> Subject: Re: [thredds] announce stable release 4.3.19 >> >> the gradle stuff is not ready to use yet - expect it will require IntelliJ >> 13. WE are still using maven, so you should "import maven project". >> >> On 10/17/2013 1:11 PM, H. Joe Lee wrote: >>> Hi, John! >>> >>> I tried to build the new Thredds but have an issue with >>> IntelliJ/Gradle on my Windows 7. >>> >>> Would you please take a look at my screenshot and tell me what could be >>> wrong? >>> >>> I git pulled the source from the documentation link that you sent. >>> >>> -- >>> HDF: Software that Powers Science >>> >>> >>> On Tue, Oct 8, 2013 at 5:26 PM, John Caron <address@hidden> wrote: >>>> The Netcdf-Java library and TDS stable release 4.3.19 is now ready: >>>> >>>> >>>> http://www.unidata.ucar.edu/software/thredds/current/netcdf-java/documentation.htm >>>> >>>> http://www.unidata.ucar.edu/software/thredds/current/tds/TDS.html >>>> >>>> (note that the above URLs are now the standard way to get the stable >>>> releases. Please update your bookmarks) >>>> >>>> This is entirely a (minor) bug fix. >>>> >>>> >>>> CDM >>>> * error assigning gregorian calendar as default to CF files >>>> * grib: protect against null vertical level name >>>> * grib time partition : ignore groups not in canonical partition >>>> * grib: fnmoc tables, try to improve >>>> * hdf5: deep error in FractalHeap >>>> * ncml: was ignoring separators in <values> tag >>>> >>>> TDS >>>> * ncss fixes >>>> * updated ncWms version (20130918) >>>> >>>> >>>> thanks! >>>> >>>> _______________________________________________ >>>> thredds mailing list >>>> address@hidden >>>> For list information or to unsubscribe, visit: >>>> http://www.unidata.ucar.edu/mailing_lists/ -- Ethan Davis UCAR Unidata Program address@hidden http://www.unidata.ucar.edu
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.