[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[THREDDS #AOP-269354]: Issue with loading GFS data set



Greetings Andy,

Sorry for the delay. I will be taking a look at this issue today and tomorrow.

Sean

> Hi Sean,
> 
> Hope you are well and safe in these times.
> 
> I was wondering what a timeline for fixing bug 
> https://github.com/Unidata/tds/issues/54 might look like. We are getting 
> closer to establishing an ERDDAP prototype and whether we can use THREDDS to 
> serve the NWP data to it or not will need a lasting architectural decision 
> soon. Is there a plan to look into this?
> 
> Kind regards,
> Andy
> 
> 
> 
> [cid:cdf50c11-0623-4ef0-b122-8c9867d9face]
> Andy Ziegler
> Strategic Data Manager â Information Services
> MetService â Te Ratonga Tirorangi
> 
> d +64 4 4700 765    t +64 4 4700 700    m +64 21 025 37630
> 30 Salamanca Road, Kelburn, Wellington 6012
> PO Box 722, Wellington 6140, New Zealand
> 
> [cid:18b9f3a8-9b67-4c79-88b8-f5d02ca7f4a3]
> 
> metservice.com<http://www.metservice.com/about/about>  .  
> metraweather.com<http://www.metraweather.com/>  .  
> metocean.co.nz<http://www.metocean.co.nz/>
> 
> MetraWeather (Australia) Pty Ltd [ACN 126 850 904], MetraWeather (UK) Ltd 
> [No. 04833498] and MetraWeather (Thailand) Ltd [No. 0105558115059] are 
> wholly-owned subsidiaries of Meteorological Service of New Zealand Ltd 
> (MetService)
> 
> 
> ________________________________
> From: Andy Ziegler
> Sent: Tuesday, November 26, 2019 3:19 PM
> To: address@hidden <address@hidden>
> Subject: RE: [THREDDS #AOP-269354]: Issue with loading GFS data set
> 
> Hi Sean,
> 
> Thank you very much for this update. Very helpful background information and 
> thank you for putting the effort in clarifying where the "issues" lie. Much 
> appreciated.
> 
> I'll keep an eye on the GitHub issue but it would be great if you ping me 
> once its fixed and a new release is out for me to test.
> I'm keen on setting up a prototype of ERDDAP and THREDDS for New Zealand 
> MetService and demonstrate how this couple could delivery on our needs for a 
> catalogue of the variety of data sources we use as well as work with model 
> forecast files as well which ERDDAP cannot do natively in an easy way. This 
> issue currently stops me from connecting them together.
> 
> Thank you for your help.
> 
> Kind regards,
> Andy
> 
> 
> -----Original Message-----
> From: Unidata THREDDS Support <address@hidden>
> Sent: Tuesday, 26 November 2019 8:59 AM
> To: Andy Ziegler <address@hidden>
> Cc: address@hidden
> Subject: [THREDDS #AOP-269354]: Issue with loading GFS data set
> 
> Greetings Andy,
> 
> There are a few issues raised in Bob's email, but they are unrelated. The 
> first issue raised is a question about the compliance of the TwoD dataset to 
> the CF Metadata Conventions, and the second is related to the use of a 
> multidimensional variable in a DAP2 Map for a Grid. These two are totally 
> unrelated, as DAP2 does not depend on the CF Metadata Conventions at all.
> 
> I've opened a github issue with regards to the CF question:
> 
> https://github.com/Unidata/netcdf-java/issues/152
> 
> Strictly speaking, there isn't a CF compliance issue, as the dataset 
> technically conforms to the CF Conventions. As described in the Overview of 
> the the CF documentation 
> (http://cfconventions.org/Data/cf-conventions/cf-conventions-1.7/cf-conventions.html#_overview),
> 
> "The use of coordinate variables is required for all dimensions that 
> correspond to one dimensional space or time coordinates. In cases where 
> coordinate variables are not applicable, the variables containing coordinate 
> data are identified by the coordinates attribute."
> 
> Since the "time" coordinate in these variables cannot be one dimensional 
> (that is, the valid time of a unique forecast is dependent the the unique 
> runtime of a given model), the second method for identifying the coordinate 
> variable is needed (i.e. inspecting the coordinates attribute).  However, it 
> is "recommended" that a multidimensional coordinate variable not share the 
> same name as a dimension, but not strictly "required" 
> (http://cfconventions.org/Data/cf-conventions/cf-conventions-1.7/cf-conventions.html#coordinate-system).
>  As the github issue above describes, we'll change our GRIB collection code 
> such that we follow the recommended practice in order to support clients that 
> do have all of the nuances of the specification covered.
> 
> I think the real issue here is that the dap2 server in the TDS is pointing to 
> a multidimensional variable in the Maps list of all of the Grids in the TwoD 
> dataset, and that is not supported by the dap2 spec (dap4 spec, yes, but not 
> dap2), which was pointed out in Bob's email (although not related to CF in 
> any way). It's likely that we'll need a code change in the dap2 server used 
> by the TDS to handle the case where individual Maps for a Grids cannot be 
> created based on name matching alone due to the violation of the 1D 
> constraint placed on a map (basically, it's a data model translation issue - 
> netCDF-Java's Common Data Model to the DAP2 Data Model). This issue is 
> outlined on github here:
> 
> https://github.com/Unidata/tds/issues/54
> 
> That's where things stand at this point. I can ping you once a fix is in 
> place, or you can follow along via github (the one to watch would be TDS 
> issue 54).
> 
> Cheers,
> 
> Sean
> 
> > Hi Ryan,
> >
> >
> >
> > Bob provided the response below. The main issue sees to be the declaration 
> > of the second dimension as a 2D field although being one of two dimensions, 
> > so should be only 1D.
> >
> > Hope this helps.
> >
> >
> >
> > Kind regards,
> >
> > Andy
> >
> >
> >
> > The response from THREDDS that I think is incorrect can be seen in the URL 
> > that you gave me:
> >
> > https://thredds.ucar.edu/thredds/dodsC/grib/NCEP/GFS/Global_0p25deg/TwoD.dds
> >
> > The part that I think is wrong is wrong for every multidimensional variable 
> > shown there, e.g.,
> >
> > Grid {
> >
> > ARRAY:
> >
> > Float32 Total_ozone_entire_atmosphere_single_layer[reftime = 14][time = 
> > 93][lat = 721][lon = 1440];
> >
> > MAPS:
> >
> > Float64 reftime[reftime = 14];
> >
> > Float64 time[reftime = 14][time = 93];
> >
> > Float32 lat[lat = 721];
> >
> > Float32 lon[lon = 1440];
> >
> > } Total_ozone_entire_atmosphere_single_layer;
> >
> >
> >
> > Specifically, I think
> >
> > Float64 time[reftime = 14][time = 93];
> >
> > should be
> >
> > Float64 time[time = 93];
> >
> >
> >
> > Multidimensional variables in CF-compliant datasets are defined by a set of 
> > 1D dimensions. Those dimensions often/usually have an associated 1D 
> > coordinate variable with the same name (e.g., a 1D time variable that just 
> > uses the 1D time dimension) -- or they can simply be dimensions without any 
> > associated coordinate variable. Section 5 of the CF standard 
> > (http://cfconventions.org/Data/cf-conventions/cf-conventions-1.7/cf-conventions.html#coordinate-system)
> >  says
> >
> >
> >
> > Any of a variableâs dimensions that is an independently varying latitude, 
> > longitude, vertical, or time dimension (see Section 1.2, "Terminology") and 
> > that has a size greater than one must have a corresponding coordinate 
> > variable, i.e., a one-dimensional variable with the same name as the 
> > dimension (see examples in Chapter 4, Coordinate Types). This is the only 
> > method of associating dimensions with coordinates that is supported by 
> > [COARDS].
> >
> >
> >
> > This use of the same name for the 1D dimension and for the 1D variable that 
> > uses only that 1D dimension is COARDS and CF's way of associating the 
> > dimension (which says "there are n values") and the actual values in the 
> > coordinate variable (which says "here are the n values").
> >
> >
> >
> > I admit that the CF specification could be clearer about this. Perhaps this 
> > use of "time" as the name of a 1D dimension and a 2D coordinate variable is 
> > legal. (It's never good when the exact meaning of a specification is open 
> > to interpretation.) But even if that is the case, it is unnecessary and 
> > confusing. It would be better if this dataset had 2 dimensions (reftime1 
> > and time), one 1D coordinate variable (reftime1[reftime1]) and a separate 
> > 2D variable with a different name which has the time value associated with 
> > each combination of [reftime1] and [time] (e.g., comboTime[reftime1][time]).
> >
> >
> >
> >
> >
> > -----Original Message-----
> > From: Unidata THREDDS Support <address@hidden>
> > Sent: Wednesday, 13 November 2019 8:55 AM
> > To: Andy Ziegler <address@hidden>
> > Cc: address@hidden
> > Subject: [THREDDS #AOP-269354]: Issue with loading GFS data set
> >
> >
> >
> > Hi,
> >
> >
> >
> > How are you pointing ERDDAP at thredds.ucar.edu? As far as I understand 
> > ERDDAP is server software, so that means you've configured a server 
> > somewhere to point to that opendap endpoint on thredds.ucar.edu?
> >
> >
> >
> > We know Bob, and can reach out as necessary, but I'm still not quite sure 
> > what the issue is or where it's coming from. Even doing:
> >
> >
> >
> > wget 
> > https://thredds.ucar.edu/thredds/dodsC/grib/NCEP/GFS/Global_0p25deg/TwoD.dds
> >
> >
> >
> > works fine for me.
> >
> >
> >
> > Ryan
> >
> >
> >
> > > Hi Ryan,
> >
> > > Thanks for looking into this.
> >
> > > Yes, I just tried it again to discover this data set with ERDDAP and get
> >
> > > *** EDDGridFromDap.generateDatasetsXml
> >
> > > tLocalSourceUrl=https://thredds.ucar.edu/thredds/dodsC/grib/NCEP/GFS/Global_0p25deg/TwoD
> >
> > > dimNames=null reloadEveryNMinutes=-1
> >
> > > externalAddGlobalAttributes=null
> >
> > > getDAS try#0 (timeout is 5 minutes)
> >
> > > getDDS try#0 (timeout is 5 minutes)
> >
> > > Error while getting DDS from 
> > > https://thredds.ucar.edu/thredds/dodsC/grib/NCEP/GFS/Global_0p25deg/TwoD.dds
> > >  .
> >
> > > In the dataset descriptor object:
> >
> > > `Total_ozone_entire_atmosphere_single_layer' is not a valid declaration. 
> > > (dods.dap.BadSemanticsException: In grid 
> > > 'Total_ozone_entire_atmosphere_single_layer The size of dimension 1 in 
> > > the array component 'Total_ozone_entire_atmosphere_single_layeris not 
> > > equal to the size of the coresponding map vector 'time2.)
> >
> > >
> >
> > > Not sure what ERDDAP uses to request the structure information of the 
> > > dataset but the error response seems to come from the TDS since Bob 
> > > Simons found the error message in the TDS code I believe (including the 
> > > typo).
> >
> > >
> >
> > > Do you know Bob and can get in touch with him if required or would you 
> > > like me to make the introduction?
> >
> >
> >
> >
> >
> > Ticket Details
> >
> > ===================
> >
> > Ticket ID: AOP-269354
> >
> > Department: Support THREDDS
> >
> > Priority: Normal
> >
> > Status: Closed
> >
> > ===================
> >
> > NOTE: All email exchanges with Unidata User Support are recorded in the 
> > Unidata inquiry tracking system and then made publicly available through 
> > the web.  If you do not want to have your interactions made available in 
> > this way, you must let us know in each email you send to us.
> >
> >
> >
> >
> >
> >
> > This email and any attachments may contain confidential information. If you 
> > are not the intended recipient, your use or communication of the 
> > information is strictly prohibited. If you have received this message in 
> > error please notify MetService immediately.
> >
> 
> 
> Ticket Details
> ===================
> Ticket ID: AOP-269354
> Department: Support THREDDS
> Priority: High
> Status: Open
> ===================
> NOTE: All email exchanges with Unidata User Support are recorded in the 
> Unidata inquiry tracking system and then made publicly available through the 
> web.  If you do not want to have your interactions made available in this 
> way, you must let us know in each email you send to us.
> 
> 
> 
> 


Ticket Details
===================
Ticket ID: AOP-269354
Department: Support THREDDS
Priority: Critical
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.



NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.