For what it's worth; from what I've seen *encoding errors* would result in
a reject from tomcat before the request is passed to the TDS application,
and it would then probably not show up in the threddsServlet.log.
Perhaphs Tom, you could try to just change the application code in your
docker image (extract war or extracted thredds servlet from one version and
then inject it in the other). I guess this should be possible, although I
have not looked into the official docker images for thredds. If you make
this fly it would be possible to say if it is the servlet or the other
libraries in the docker image that causes you problems.
Regarding the possible encoding errors: Tomcat introduced the more strict
behavior in v9.0.8 (and also v8.5.31, v8.0.52 and v7.0.87).
See the corresponding tomcat changelogs:
- [image: Add:] 62273
<http://bz.apache.org/bugzilla/show_bug.cgi?id=62273>: Implement
configuration options to work-around specification non-compliant user
agents (including all the major browsers) that do not correctly %nn encode
URI paths and query strings as required by RFC 7230 and RFC 3986. (markt)
See this thread in the google group for opendap.org:
https://groups.google.com/a/opendap.org/forum/#!msg/support/ixTqhDXoLZQ/IT0lvZQ7CAAJ
Tomcat-documentation:
https://tomcat.apache.org/tomcat-9.0-doc/config/http.html
relaxedPathChars
The HTTP/1.1 specification <https://tools.ietf.org/rfc/rfc7230.txt> requires
that certain characters are %nn encoded when used in URI paths.
Unfortunately, many user agents including all the major browsers are not
compliant with this specification and use these characters in unencoded
form. To prevent Tomcat rejecting such requests, this attribute may be used
to specify the additional characters to allow. If not specified, no
addtional characters will be allowed. The value may be any combination of
the following characters: " < > [ \ ] ^ ` { | } . Any other characters
present in the value will be ignored.
relaxedQueryChars
The HTTP/1.1 specification <https://tools.ietf.org/rfc/rfc7230.txt> requires
that certain characters are %nn encoded when used in URI query strings.
Unfortunately, many user agents including all the major browsers are not
compliant with this specification and use these characters in unencoded
form. To prevent Tomcat rejecting such requests, this attribute may be used
to specify the additional characters to allow. If not specified, no
addtional characters will be allowed. The value may be any combination of
the following characters: " < > [ \ ] ^ ` { | } . Any other characters
present in the value will be ignored.
As the OPeNDAP spesification (see section 6.1.1.2 on page 22)
<https://cdn.earthdata.nasa.gov/conduit/upload/512/ESE-RFC-004v1.1.pdf>
requires
the use of [ and ] in the query string, and several clients doesn't quote
these characters, we need to tweak the relaxedQueryChars configuration for
tomcat.
Regards,
Christian Skarby
Den tor. 17. jan. 2019 kl. 05:42 skrev Thomas Cook <tmcook@xxxxxxxx>:
> I do not know how to debug the matlab or python clients, but this is
> what shows up in the threddsServlet.log
>
> Request: "GET
> /thredds/dodsC/HFR/USWC/6km/hourly/RTV/HFRADAR_US_West_Coast_6km_Resolution_Hourly_RTV_best.ncd.dods?lat,lon,site%5flat,site%5flon,site%5fcode,site%5fnetCode,procParams
> HTTP/1.0"
>
> Thanks
> Tom
>
> On Wed, Jan 16, 2019 at 7:47 PM Dennis Heimbigner <dmh@xxxxxxxx> wrote:
> >
> > Is there anyway you can see the exact URL being sent to the server?
> >
> > On 1/16/2019 8:24 PM, Thomas Cook wrote:
> > > Thanks Sean.
> > > The server is http://hfrnet-tds.ucsd.edu/thredds (although I just
> > > downgraded back to 4.6.12-SNAPSHOT). I will see about making my test
> > > server accessible to you, but that will take a little time to
> > > configure on my end.
> > > Let me know if you think I should open an issue on the github page and
> > > move the discussion there.
> > > Tom
> > >
> > > On Wed, Jan 16, 2019 at 6:32 PM Sean Arms <sarms@xxxxxxxx> wrote:
> > >> Hi Tom,
> > >>
> > >> So you are hitting up an opendap endpoint using all three libraries /
> tools (python, matlab, and panoply)? Is this particular server publicly
> accessible?
> > >>
> > >> I have a bad feeling that this is due to URL encoding (or lack
> thereof) on the client side and changes made in tomcat 8.5, which it looks
> like the thredds-docker repository bumped up to with the 4.6.12 release.
> > >>
> > >> Cheers,
> > >>
> > >> Sean
> > >>
> > >>
> > >> On Wed, Jan 16, 2019 at 7:06 PM Thomas Cook <tmcook@xxxxxxxx> wrote:
> > >>> HI I just wanted to add some details.
> > >>>
> > >>> The functionality works when I downgraded to the docker container
> > >>> 4.6.12-SNAPSHOT
> > >>>
> > >>> My data is part of a FMRC collection.
> > >>>
> > >>> Thanks,
> > >>> Tom
> > >>>
> > >>> On Wed, Jan 16, 2019 at 4:51 PM Thomas Cook <tmcook@xxxxxxxx> wrote:
> > >>>> HI
> > >>>> I just yesterday updated my thredds-docker to 4.6.12 from
> > >>>> 4.6.12-snapshot. It seems to be running fine, except I'm getting
> > >>>> errors using matlab ncread and python netcdf4.dataset. When I try to
> > >>>> access any of the multi-dimensional (time, lat, lon) variables I get
> > >>>> the following errors
> > >>>> with matlab:
> > >>>> Error using netcdflib
> > >>>> The NetCDF library encountered an error during execution of
> > >>>> 'getVarsShort' function - 'Access failure (-77)'.
> > >>>>
> > >>>> Error in netcdf.getVar (line 136)
> > >>>> data = netcdflib(funcstr,ncid,varid,varargin{:});
> > >>>>
> > >>>> Error in internal.matlab.imagesci.nc/read (line 635)
> > >>>> data = netcdf.getVar(gid, varid, ...
> > >>>>
> > >>>> Error in ncread (line 58)
> > >>>> vardata = ncObj.read(varName, varargin{:});
> > >>>>
> > >>>> Error in readUVfromTDS (line 56)
> > >>>> u=ncread(url, 'u_mean', [1 1 1], [2 2 2]);
> > >>>>
> > >>>>
> > >>>> and with python:
> > >>>> netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable.__getitem__()
> > >>>>
> > >>>> netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable._get()
> > >>>>
> > >>>> netCDF4/_netCDF4.pyx in netCDF4._netCDF4._ensure_nc_success()
> > >>>>
> > >>>> RuntimeError: NetCDF: Access failure
> > >>>>
> > >>>> I know that the latest docker contains netcdf 4.6, where as the
> > >>>> snapshot had 4.4. Could it be a version problem with the netCD4
> python
> > >>>> and matlab libraries? I should be running the latest netcdf python
> > >>>> version and not too far behind with matlab versions.
> > >>>>
> > >>>> The weird things are that the index variables are accessible (time,
> > >>>> lat, lon etc), and the multiple-dimension data is accessible just
> > >>>> fine using the OpenDap web interface and Panoply. So I'm a little
> > >>>> confused. I'm addressing this list since I think there might be some
> > >>>> matlab and python expertise here and I think there were some
> messages
> > >>>> about the Access failure error in the past.
> > >>>>
> > >>>> Thanks,
> > >>>> Tom
> > >>> _______________________________________________
> > >>> NOTE: All exchanges posted to Unidata maintained email lists are
> > >>> recorded in the Unidata inquiry tracking system and made publicly
> > >>> available through the web. Users who post to any of the lists we
> > >>> maintain are reminded to remove any personal information that they
> > >>> do not want to be made public.
> > >>>
> > >>>
> > >>> thredds mailing list
> > >>> thredds@xxxxxxxxxxxxxxxx
> > >>> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
> > > _______________________________________________
> > > NOTE: All exchanges posted to Unidata maintained email lists are
> > > recorded in the Unidata inquiry tracking system and made publicly
> > > available through the web. Users who post to any of the lists we
> > > maintain are reminded to remove any personal information that they
> > > do not want to be made public.
> > >
> > >
> > > thredds mailing list
> > > thredds@xxxxxxxxxxxxxxxx
> > > For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
> >
> > _______________________________________________
> > NOTE: All exchanges posted to Unidata maintained email lists are
> > recorded in the Unidata inquiry tracking system and made publicly
> > available through the web. Users who post to any of the lists we
> > maintain are reminded to remove any personal information that they
> > do not want to be made public.
> >
> >
> > thredds mailing list
> > thredds@xxxxxxxxxxxxxxxx
> > For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> thredds mailing list
> thredds@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
>
