[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[THREDDS #QCX-921252]: Wrong dates reported by thredds 4.3



Rostislav,

I've been thinking about your question, and I'm afraid I could not come up with 
a mechanism to add the variables you mention to a netcdf subset.  
Unfortunately, for reasons to do with CF imlementation requirements, this is 
not a feature we support (though we may implement this in a future release).  
Essentially, if you were serving up individual files, it would be no problem to 
wrap the files in a bit of NcML to add the dimension and variables.  However, 
as an aggregation, the best solution I can offer is to add the information as a 
global attribute, like this:

  <dataset name="Rostislav">
    <featureCollection name="Rostislavnc4data" featureType="FMRC" 
harvest="true" path="rostislav/fmrc">
      <metadata inherited="true">
        <serviceName>all</serviceName>
        <dataFormat>netCDF</dataFormat>
      </metadata>
      <collection spec="C:/Users/madry/work/RostislavKouznetsov/.*nc4" />
      <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
        <dimension name="dayhours" value="24"/>
        <attribute name="dayhours" value="24"/>
      </netcdf>
    </featureCollection>
  </dataset>

It may be worth noting that the information you would like to add is more akin 
to metadata than actual data.

Regards,
  Lansing Madry
  Unidata
  Boulder, Colorado

> Hello Lansing,
> 
> Thank you for ther reply.
> On the second point.
> If you download a file from
> http://silam.fmi.fi/thredds/fileServer/fires/files/reanal_v1_3_global_2000.nc4
> you will find there a dimension
> dayhours = 24 ;
> and two variables
> float dayhours(dayhours) ;
> float hourlyadj(dayhours) ;
> 
> They do not appear in files that come from netcdfsubset
> http://silam.fmi.fi/thredds/ncss/grid/fires/files/reanal_v1_3_global_2000.nc4?var=emis_PM_FRP&north=70&west=0&east=20&south=50&horizStride=1&time_start=2000-01-01T00%3A00%3A00Z&time_end=2000-12-31T00%3A00%3A00Z&timeStride=1&addLatLon=true&accept=netcdf
> 
> Is it possible to make the dimension "dayhours" and correspondinf
> variables to appear in netcdf subsets?
> 
> Thank you once more.
> 
> Best regards,
> 
> Rostislav
> 
> 
> 
> On Sat, Apr 20, 2013 at 01:09:30AM +0300, Unidata THREDDS Support wrote:
> >Rostislav,
> >
> >I was able to reproduce the failure you saw when asking for NetCDF4 versus 
> >NetCDF files.  I wasn't sure why that was happening, so I asked the 
> >developer who worked on the ncss service.  He thought it was probably a bug 
> >of some sort, so I opened a ticket to address the issue.  I don't know if it 
> >is an easy fix or not yet, but it is in the queue.  Thank you for bringing 
> >this to our attention!
> >
> >https://bugtracking.unidata.ucar.edu/browse/TDS-408
> >
> >On your second point, I'm not sure what you mean by the small, non-grid 
> >variable.  When I do an ncdump on the file, I see:
> >
> >  dimensions:
> >    lon = 3600;
> >    lat = 1800;
> >    time = UNLIMITED;   // (1 currently
> >  variables:
> >    float lon(lon=3600);
> >      :long_name = "longitude";
> >      :units = "degrees_east";
> >      :standard_name = "longitude";
> >
> >    float lat(lat=1800);
> >      :units = "degrees_north";
> >      :standard_name = "latitude";
> >      :long_name = "latitude";
> >
> >    double time(time=1);
> >      :units = "seconds since 1970-01-01 00:00:00";
> >      :calendar = "standard";
> >      :_ChunkSize = 1; // int
> >
> >    float emis_PM_FRP(time=1, lat=1800, lon=3600);
> >      :units = "kg/s";
> >      :_ChunkSize = 1, 1800, 3600; // int
> >
> >  // global attributes:
> >  :CDI = "Climate Data Interface version 1.4.1";
> >  :Conventions = "CF-1.0";
> >  :netcdf_generated = "2013-04-15 09:01 UTC";
> >  :producer = "Finnish Meteorological Institute";
> >  :data_produced = "2013-04-13";
> >  :dataset = "IS4FIRES wildland-fire emission reanalysis";
> >  :contact = "Mikhail Sofiev: address@hidden";
> >  :PM2P5_10fraction = 0.25999999046325684; // double
> >  :PM2P5fraction = 0.7400000095367432; // double
> >  :quantity = "PM emission rate";
> >  :CDO = "Climate Data Operators version 1.4.1 
> > (http://www.mpimet.mpg.de/cdo)";
> >
> >Which is the non-grid variable you mean?
> >
> >On the third point, I will have to see if there is a way to get the 
> >information you want served up in fewer steps, but I don't know if it is 
> >possible off-hand.
> >
> >Regards,
> >  Lansing
> >> Hello Lansing,
> >>
> >> Thank you for the detailed reply.
> >> I have set up the service according to your instructions.
> >> http://silam.fmi.fi/thredds/firesEmission.html
> >> Few more questions.
> >>
> >> 1. The subsetting of files does not work with nc4 output
> >> when accessing by file:
> >> http://silam.fmi.fi/thredds/ncss/grid/fires/files/reanal_v1_3_global_2000.nc4?var=emis_PM_FRP&time_start=2000-01-01T00%3A00%3A00Z&time_end=2000-01-02T00%3A00%3A00Z&accept=netcdf
> >>  works, but
> >> http://silam.fmi.fi/thredds/ncss/grid/fires/files/reanal_v1_3_global_2000.nc4?var=emis_PM_FRP&time_start=2000-01-01T00%3A00%3A00Z&time_end=2000-01-02T00%3A00%3A00Z&accept=netcdf4
> >>  (difference in the last argument)
> >> causes "I/O Exception handled : NetCDF: Invalid argument
> >> nc_def_var_chunking on variable time"
> >> Is it a bug or a feature?
> >>
> >> 2. I have a small non-grid variable in my file. It disappears when
> >> accessing the files via thredds (except for http).
> >> Can i make thredds to include it  into any netcdf produced
> >> from the datasets?
> >>
> >> 3. The data are obtained from reanalisys, so they have a single
> >> time dimension. To get to the metadata page for "Fire emission this year"
> >> i have to click "Fire emission this year" and then "Best timeseries".
> >> Is it possible to avoid the second step?
> >>
> >>
> >> Thank you once more!
> >>
> >> Best regards,
> >> Rostislav
> >>
> >>
> >> On Wed, Apr 17, 2013 at 11:49:39PM +0300, Unidata THREDDS Support wrote:
> >> >Hello Rostislav,
> >> >
> >> >I downloaded a few days worth of files from your TDS and experimented 
> >> >with setting up a local TDS to help with your questions.  To answer your 
> >> >first question, what you are observing is a feature, but I think you are 
> >> >misinterpreting the data.  With a datasetScan such as you have, the time 
> >> >reported is the last time the file was modified (i.e., touched on your 
> >> >server), not the time or date information contained in the file.  To 
> >> >answer your second question, I believe what you really want to do is to 
> >> >set up a featureCollection, instead of a datasetScan.  Here is a simple 
> >> >example that I used locally:
> >> >
> >> ><?xml version="1.0" encoding="UTF-8"?>
> >> ><catalog name="THREDDS Server Default Catalog : You must change this to 
> >> >fit your server!"
> >> >        
> >> > xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0";
> >> >        xmlns:xlink="http://www.w3.org/1999/xlink";>
> >> >
> >> >  <service name="all" base="" serviceType="compound">
> >> >    <service name="odap" serviceType="OpenDAP" base="/thredds/dodsC/" />
> >> >    <service name="http" serviceType="HTTPServer" 
> >> > base="/thredds/fileServer/" />
> >> >    <!--service name="wcs" serviceType="WCS" base="/thredds/wcs/" /-->
> >> >    <service name="wms" serviceType="WMS" base="/thredds/wms/" />
> >> >    <service name="ncss" serviceType="NetcdfSubset" 
> >> > base="/thredds/ncss/grid/" />
> >> >  </service>
> >> >
> >> >  <dataset name="Rostislav">
> >> >    <featureCollection name="Rostislavnc4data" featureType="FMRC" 
> >> > harvest="true" path="rostislav/fmrc">
> >> >      <metadata inherited="true">
> >> >        <serviceName>all</serviceName>
> >> >        <dataFormat>netCDF</dataFormat>
> >> >      </metadata>
> >> >      <collection spec="C:/Users/madry/work/RostislavKouznetsov/.*nc4" />
> >> >    </featureCollection>
> >> >  </dataset>
> >> ></catalog>
> >> >
> >> >This sets your data up to be served as a single logical unit via the 
> >> >NetCDF Subset Service.  After you have changed the names and paths to 
> >> >suit your system, you will also need to edit the threddsConfig.xml file 
> >> >to turn the NetcdfSubsetService on.  You do this by uncommenting the 
> >> >relevant block of xml and setting <allow> to true:
> >> >
> >> >  <!--
> >> >  The Netcdf Subset Service is off by default.
> >> >  -->
> >> >  <NetcdfSubsetService>
> >> >    <allow>true</allow>
> >> >    <scour>10 min</scour>
> >> >    <maxAge>-1 min</maxAge>
> >> >  </NetcdfSubsetService>
> >> >
> >> >Let me know if you need any further assistance.
> >> >
> >> >Regards,
> >> >  Lansing Madry
> >> >  Unidata
> >> >  Boulder, Colorado
> >> >
> >> >> Hello,
> >> >>
> >> >> I have put a collection of daily files into thredds, one field per day.
> >> >> http://silam.fmi.fi/thredds/catalog/fires3/catalog.html
> >> >>
> >> >> Unfortunately it reports wrong times for them.  e.g.
> >> >>
> >> >> http://silam.fmi.fi/thredds/catalog/fires3/catalog.html?dataset=fires3/reanal_v1_3_global_20130410.nc4
> >> >>
> >> >> shows "date 2013-04-15T09:01:19Z (modified)" instead of something like
> >> >> 2013-04-10T00:00:00Z.
> >> >>
> >> >> The dataset is described as:
> >> >> <datasetScan name="Fires3" path="fires3"
> >> >> location="/data/thredds/Fires/"  suffix=".nc4">
> >> >> <serviceName>all</serviceName>
> >> >> </datasetScan>
> >> >>
> >> >> Is it a bug or a feature?
> >> >>
> >> >> Can I somehow configure thredds to report these files as a single
> >> >> dataset?
> >> >>
> >> >> Thank you!
> >> >>
> >> >> Best regards,
> >> >> Rostislav
> >> >>
> >> >> --
> >> >> Rostislav Kouznetsov
> >> >>
> >> >> **************************************
> >> >> Air quality
> >> >> Finnish Meteorological Institute
> >> >> PL-503 FI-00101 Helsinki Finland
> >> >> Phone: +358 29 539 4630
> >> >> **************************************
> >> >>
> >> >>
> >> >
> >> >
> >> >Ticket Details
> >> >===================
> >> >Ticket ID: QCX-921252
> >> >Department: Support THREDDS
> >> >Priority: Normal
> >> >Status: Open
> >> >
> >>
> >> --
> >> Rostislav Kouznetsov
> >>
> >> **************************************
> >> Air quality
> >> Finnish Meteorological Institute
> >> PL-503 FI-00101 Helsinki Finland
> >> Phone: +358 29 539 4630
> >> **************************************
> >>
> >>
> >
> >
> >Ticket Details
> >===================
> >Ticket ID: QCX-921252
> >Department: Support THREDDS
> >Priority: Normal
> >Status: Open
> >
> 
> --
> Rostislav Kouznetsov
> 
> **************************************
> Air quality
> Finnish Meteorological Institute
> PL-503 FI-00101 Helsinki Finland
> Phone: +358 29 539 4630
> **************************************
> 
> 


Ticket Details
===================
Ticket ID: QCX-921252
Department: Support THREDDS
Priority: High
Status: Open


NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.