[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[THREDDS #AOP-269354]: Issue with loading GFS data set

Greetings Andy,

There are a few issues raised in Bob's email, but they are unrelated. The first 
issue raised is a question about the compliance of the TwoD dataset to the CF 
Metadata Conventions, and the second is related to the use of a 
multidimensional variable in a DAP2 Map for a Grid. These two are totally 
unrelated, as DAP2 does not depend on the CF Metadata Conventions at all.

I've opened a github issue with regards to the CF question:

https://github.com/Unidata/netcdf-java/issues/152

Strictly speaking, there isn't a CF compliance issue, as the dataset 
technically conforms to the CF Conventions. As described in the Overview of the 
the CF documentation 
(http://cfconventions.org/Data/cf-conventions/cf-conventions-1.7/cf-conventions.html#_overview),

"The use of coordinate variables is required for all dimensions that correspond 
to one dimensional space or time coordinates. In cases where coordinate 
variables are not applicable, the variables containing coordinate data are 
identified by the coordinates attribute."

Since the "time" coordinate in these variables cannot be one dimensional (that 
is, the valid time of a unique forecast is dependent the the unique runtime of 
a given model), the second method for identifying the coordinate variable is 
needed (i.e. inspecting the coordinates attribute).  However, it is 
"recommended" that a multidimensional coordinate variable not share the same 
name as a dimension, but not strictly "required" 
(http://cfconventions.org/Data/cf-conventions/cf-conventions-1.7/cf-conventions.html#coordinate-system).
 As the github issue above describes, we'll change our GRIB collection code 
such that we follow the recommended practice in order to support clients that 
do have all of the nuances of the specification covered.

I think the real issue here is that the dap2 server in the TDS is pointing to a 
multidimensional variable in the Maps list of all of the Grids in the TwoD 
dataset, and that is not supported by the dap2 spec (dap4 spec, yes, but not 
dap2), which was pointed out in Bob's email (although not related to CF in any 
way). It's likely that we'll need a code change in the dap2 server used by the 
TDS to handle the case where individual Maps for a Grids cannot be created 
based on name matching alone due to the violation of the 1D constraint placed 
on a map (basically, it's a data model translation issue - netCDF-Java's Common 
Data Model to the DAP2 Data Model). This issue is outlined on github here:

https://github.com/Unidata/tds/issues/54

That's where things stand at this point. I can ping you once a fix is in place, 
or you can follow along via github (the one to watch would be TDS issue 54).

Cheers,

Sean

> Hi Ryan,
> 
> 
> 
> Bob provided the response below. The main issue sees to be the declaration of 
> the second dimension as a 2D field although being one of two dimensions, so 
> should be only 1D.
> 
> Hope this helps.
> 
> 
> 
> Kind regards,
> 
> Andy
> 
> 
> 
> The response from THREDDS that I think is incorrect can be seen in the URL 
> that you gave me:
> 
> https://thredds.ucar.edu/thredds/dodsC/grib/NCEP/GFS/Global_0p25deg/TwoD.dds
> 
> The part that I think is wrong is wrong for every multidimensional variable 
> shown there, e.g.,
> 
> Grid {
> 
> ARRAY:
> 
> Float32 Total_ozone_entire_atmosphere_single_layer[reftime = 14][time = 
> 93][lat = 721][lon = 1440];
> 
> MAPS:
> 
> Float64 reftime[reftime = 14];
> 
> Float64 time[reftime = 14][time = 93];
> 
> Float32 lat[lat = 721];
> 
> Float32 lon[lon = 1440];
> 
> } Total_ozone_entire_atmosphere_single_layer;
> 
> 
> 
> Specifically, I think
> 
> Float64 time[reftime = 14][time = 93];
> 
> should be
> 
> Float64 time[time = 93];
> 
> 
> 
> Multidimensional variables in CF-compliant datasets are defined by a set of 
> 1D dimensions. Those dimensions often/usually have an associated 1D 
> coordinate variable with the same name (e.g., a 1D time variable that just 
> uses the 1D time dimension) -- or they can simply be dimensions without any 
> associated coordinate variable. Section 5 of the CF standard 
> (http://cfconventions.org/Data/cf-conventions/cf-conventions-1.7/cf-conventions.html#coordinate-system)
>  says
> 
> 
> 
> Any of a variable’s dimensions that is an independently varying latitude, 
> longitude, vertical, or time dimension (see Section 1.2, "Terminology") and 
> that has a size greater than one must have a corresponding coordinate 
> variable, i.e., a one-dimensional variable with the same name as the 
> dimension (see examples in Chapter 4, Coordinate Types). This is the only 
> method of associating dimensions with coordinates that is supported by 
> [COARDS].
> 
> 
> 
> This use of the same name for the 1D dimension and for the 1D variable that 
> uses only that 1D dimension is COARDS and CF's way of associating the 
> dimension (which says "there are n values") and the actual values in the 
> coordinate variable (which says "here are the n values").
> 
> 
> 
> I admit that the CF specification could be clearer about this. Perhaps this 
> use of "time" as the name of a 1D dimension and a 2D coordinate variable is 
> legal. (It's never good when the exact meaning of a specification is open to 
> interpretation.) But even if that is the case, it is unnecessary and 
> confusing. It would be better if this dataset had 2 dimensions (reftime1 and 
> time), one 1D coordinate variable (reftime1[reftime1]) and a separate 2D 
> variable with a different name which has the time value associated with each 
> combination of [reftime1] and [time] (e.g., comboTime[reftime1][time]).
> 
> 
> 
> 
> 
> -----Original Message-----
> From: Unidata THREDDS Support <address@hidden>
> Sent: Wednesday, 13 November 2019 8:55 AM
> To: Andy Ziegler <address@hidden>
> Cc: address@hidden
> Subject: [THREDDS #AOP-269354]: Issue with loading GFS data set
> 
> 
> 
> Hi,
> 
> 
> 
> How are you pointing ERDDAP at thredds.ucar.edu? As far as I understand 
> ERDDAP is server software, so that means you've configured a server somewhere 
> to point to that opendap endpoint on thredds.ucar.edu?
> 
> 
> 
> We know Bob, and can reach out as necessary, but I'm still not quite sure 
> what the issue is or where it's coming from. Even doing:
> 
> 
> 
> wget 
> https://thredds.ucar.edu/thredds/dodsC/grib/NCEP/GFS/Global_0p25deg/TwoD.dds
> 
> 
> 
> works fine for me.
> 
> 
> 
> Ryan
> 
> 
> 
> > Hi Ryan,
> 
> > Thanks for looking into this.
> 
> > Yes, I just tried it again to discover this data set with ERDDAP and get
> 
> > *** EDDGridFromDap.generateDatasetsXml
> 
> > tLocalSourceUrl=https://thredds.ucar.edu/thredds/dodsC/grib/NCEP/GFS/Global_0p25deg/TwoD
> 
> > dimNames=null reloadEveryNMinutes=-1
> 
> > externalAddGlobalAttributes=null
> 
> > getDAS try#0 (timeout is 5 minutes)
> 
> > getDDS try#0 (timeout is 5 minutes)
> 
> > Error while getting DDS from 
> > https://thredds.ucar.edu/thredds/dodsC/grib/NCEP/GFS/Global_0p25deg/TwoD.dds
> >  .
> 
> > In the dataset descriptor object:
> 
> > `Total_ozone_entire_atmosphere_single_layer' is not a valid declaration. 
> > (dods.dap.BadSemanticsException: In grid 
> > 'Total_ozone_entire_atmosphere_single_layer The size of dimension 1 in the 
> > array component 'Total_ozone_entire_atmosphere_single_layeris not equal to 
> > the size of the coresponding map vector 'time2.)
> 
> >
> 
> > Not sure what ERDDAP uses to request the structure information of the 
> > dataset but the error response seems to come from the TDS since Bob Simons 
> > found the error message in the TDS code I believe (including the typo).
> 
> >
> 
> > Do you know Bob and can get in touch with him if required or would you like 
> > me to make the introduction?
> 
> 
> 
> 
> 
> Ticket Details
> 
> ===================
> 
> Ticket ID: AOP-269354
> 
> Department: Support THREDDS
> 
> Priority: Normal
> 
> Status: Closed
> 
> ===================
> 
> NOTE: All email exchanges with Unidata User Support are recorded in the 
> Unidata inquiry tracking system and then made publicly available through the 
> web.  If you do not want to have your interactions made available in this 
> way, you must let us know in each email you send to us.
> 
> 
> 
> 
> 
> 
> This email and any attachments may contain confidential information. If you 
> are not the intended recipient, your use or communication of the information 
> is strictly prohibited. If you have received this message in error please 
> notify MetService immediately.
> 


Ticket Details
===================
Ticket ID: AOP-269354
Department: Support THREDDS
Priority: High
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.