[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: aggregation of individual in situ file collections



On 5/16/2011 12:39 PM, Steve Hankin wrote: John,

At GO-ESSP you mentioned aggregation techniques that can present collections of in situ obs from individual files as virtual collections (already in use in Motherlode?).  I think I understood you to imply that these aggregations are already accessible through alpha/beta ncStream code.  (Is this right?)

If these capabilities are available we have "strategic" reasons to start playing with them in a couple of NOAA projects.  There are huge collections of in situ ocean files accumulating ... waiting for a solution like this.  Can you point us towards what's needed to test the the server-side aggregation techniques?  (I assume the client side must be embedded in the beta Java netCDF libraries, no?)

    thanks - Steve

the catalog is:

  http://motherlode.ucar.edu:8080/thredds/idd/newPointObs.html

attached is the config catalog.

the only dataset really tested is ( i think theres bugs in the others):

  http://motherlode.ucar.edu:8080/thredds/cdmrfeature/nws/metar/ncdecoded/Metar_Station_Data_fc.cdmr?req=form

a bit of  docs at:

http://www.unidata.ucar.edu/projects/THREDDS/tech/tds4.2/reference/collections/FeatureCollections.html

search for pointConfig. part of the reason its not released yet is that we dont have reliable docs yet for it. ill try to remedy that by the workshop in July.

yes, you need CDM to read, although there is CVS and XML options (eg on the form) which should be useful.

http://www.unidata.ucar.edu/software/netcdf-java/reference/FeatureDatasets/PointFeatures.html

it will be some time before i can really pay attention to this, so use as alpha where you can tolerate delays. you can use the latest TDS 4.2 release on the server.



<?xml version="1.0" encoding="UTF-8"?>
<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0";
         xmlns:xlink="http://www.w3.org/1999/xlink"; name="THREDDS-IDD OPeNDAP 
Data Server - New Station Data"
         version="1.0.3">

  <service name="latest" serviceType="Resolver" base=""/>

  <service name="all" serviceType="Compound" base="">
    <service name="cdmremote" serviceType="CdmRemote" 
base="/thredds/cdmremote/"/>
    <service name="ncdods" serviceType="OPENDAP" base="/thredds/dodsC/"/>
    <service name="HTTPServer" serviceType="HTTPServer" 
base="/thredds/fileServer/"/>
  </service>

  <service name="forBufr" serviceType="Compound" base="">
    <service name="cdmremote" serviceType="CdmRemote" 
base="/thredds/cdmremote/"/>
    <service name="HTTPServer" serviceType="HTTPServer" 
base="/thredds/fileServer/"/>
  </service>

  <dataset name="Point Feature Collections">
    <metadata inherited="true">
      <serviceName>all</serviceName>
      <authority>edu.ucar.unidata</authority>
      <dataType>Station</dataType>
      <documentation type="rights">Freely available</documentation>
      <creator>
        <name vocabulary="DIF">DOC/NOAA/NWS</name>
      </creator>
      <publisher>
        <name vocabulary="DIF">UCAR/UNIDATA</name>
      </publisher>
    </metadata>

    <featureCollection name="Metar Station Data" harvest="true" 
featureType="Station" path="nws/metar/ncdecoded">
      <metadata inherited="true">
        <documentation type="summary">Metars: hourly surface weather 
observations</documentation>
        <documentation xlink:href="http://metar.noaa.gov/"; 
xlink:title="NWS/NOAA information"/>
        <documentation>In the U.S., METAR reports are taken once an hour 
between 50 minutes past the hour and the top of
          the (next) hour. All the observations taken within this time are 
considered to be for the same cycle. For
          example, observations taken between 1150Z and 1200Z are all 
considered to be 12Z observations. In reality,
          observations may be taken a little earlier or a little later. In 
addition, observations taken due to changing
          conditions, known as SPECI reports, are also considered to occur for 
a given cycle. Our rule of thumb is that
          all reports taken between (hh-1)45Z to hh44Z are considered to be for 
the hh cycle.
        </documentation>
        <keyword>metar</keyword>
        <keyword>surface observations</keyword>
      </metadata>

      <property name="raw" value="report"/>
      <property name="resolution" value="20 min"/>

      <collection 
spec="/data/ldm/pub/decoded/netcdf/surface/metar/Surface_METAR_#yyyyMMdd_HHmm#.nc$"
 />
      <update startup="true" rescan="0 0/15 * * * ? *" trigger="allow"/>
      <protoDataset choice="Penultimate" />
      <pointConfig datasetTypes="cdmrFeature Files"/>
    </featureCollection>

    <featureCollection name="Surface Buoy Point Data" harvest="true" 
featureType="Point" path="nws/buoy/ncdecoded">
      <collection 
spec="/data/ldm/pub/decoded/netcdf/surface/buoy/Surface_Buoy_#yyyyMMdd_HHmm#.nc$"
 />
      <update startup="true" rescan="0 0/15 * * * ? *" trigger="allow"/>
      <protoDataset choice="Penultimate" />
      <pointConfig datasetTypes="cdmrFeature Files"/>
    </featureCollection>

    <featureCollection name="Surface Synoptic Point Data" harvest="true" 
featureType="Point" path="nws/synoptic/ncdecoded">
      <collection 
spec="/data/ldm/pub/decoded/netcdf/surface/synoptic/Surface_Synoptic_#yyyyMMdd_HHmm#.nc$"
 />
      <update startup="true" rescan="0 0/15 * * * ? *" trigger="allow"/>
      <protoDataset choice="Penultimate" />
      <pointConfig datasetTypes="cdmrFeature Files"/>
    </featureCollection>

    <featureCollection name="Upper Air Point Data" harvest="true" 
featureType="Point" path="nws/upperair/ncdecoded">
      <collection 
spec="/data/ldm/pub/decoded/netcdf/upperair/Upperair_#yyyyMMdd_HHmm#.nc$" />  
      <update startup="true" rescan="0 0/15 * * * ? *" trigger="allow"/>
      <protoDataset choice="Penultimate" />
      <pointConfig datasetTypes="cdmrFeature Files"/>
    </featureCollection>

  </dataset>
</catalog>


NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.