OK Given Nathan and Christian's comments, I'm focussing my
troubleshooting on client side.
What I've learned
1) Matlab's native netCDF toolbox depends on netCDF 4.3.3.1. I
couldn't find an easy way to update the netCDF version inside my
Matlab install.
2) I updated the netcdf4-python module to use netCDF 4.6.12. This did
not solve the issue.
3) Panoply (which works) is using netcdfAll-5.0.0-SNAPSHOT. I couldn't
find an older version to download, nor was I able to run panoply with
an older netcdfAll.jar
4) Another user has reported success using the SNTOOLS/MEXNC package
to read the files. This also uses a version of netcdfAll.jar for its
decoding. I can't verify this yet.
So not sure there's much more I can do with on the client side. I'll
mess with the Dockerfile like Christian suggested, and try using some
non FMRC collections to test with the clients. If anyone has any
ideas, I'm open!
Thanks to all who have replied and I'll update if I make any headway.
Tom
On Thu, Jan 17, 2019 at 5:53 AM Nathan Potter <ndp@xxxxxxxxxxx> wrote:
>
>
>
> > On Jan 17, 2019, at 5:37 AM, Christian Skarby <christians@xxxxxx> wrote:
> >
> > For what it's worth; from what I've seen encoding errors would result in a
> > reject from tomcat before the request is passed to the TDS application, and
> > it would then probably not show up in the threddsServlet.log.
>
> This is the behavior that we have seen.
>
> Tomcat's rejection is an empty document and response headers ala:
>
> HTTP/1.1 400 Bad Request
> Server: Apache-Coyote/1.1
> Transfer-Encoding: chunked
> Date: Thu, 17 Jan 2019 13:47:48 GMT
> Connection: close
>
> >
> > Perhaphs Tom, you could try to just change the application code in your
> > docker image (extract war or extracted thredds servlet from one version and
> > then inject it in the other). I guess this should be possible, although I
> > have not looked into the official docker images for thredds. If you make
> > this fly it would be possible to say if it is the servlet or the other
> > libraries in the docker image that causes you problems.
> >
> > Regarding the possible encoding errors: Tomcat introduced the more strict
> > behavior in v9.0.8 (and also v8.5.31, v8.0.52 and v7.0.87).
> >
> > See the corresponding tomcat changelogs:
> > • 62273: Implement configuration options to work-around
> > specification non-compliant user agents (including all the major browsers)
> > that do not correctly %nn encode URI paths and query strings as required by
> > RFC 7230 and RFC 3986. (markt)
> > See this thread in the google group for opendap.org:
> > https://groups.google.com/a/opendap.org/forum/#!msg/support/ixTqhDXoLZQ/IT0lvZQ7CAAJ
> >
> > Tomcat-documentation:
> > https://tomcat.apache.org/tomcat-9.0-doc/config/http.html
> >
> > relaxedPathChars
> > The HTTP/1.1 specification requires that certain characters are %nn encoded
> > when used in URI paths. Unfortunately, many user agents including all the
> > major browsers are not compliant with this specification and use these
> > characters in unencoded form. To prevent Tomcat rejecting such requests,
> > this attribute may be used to specify the additional characters to allow.
> > If not specified, no addtional characters will be allowed. The value may be
> > any combination of the following characters: " < > [ \ ] ^ ` { | } . Any
> > other characters present in the value will be ignored.
> >
> > relaxedQueryChars
> > The HTTP/1.1 specification requires that certain characters are %nn encoded
> > when used in URI query strings. Unfortunately, many user agents including
> > all the major browsers are not compliant with this specification and use
> > these characters in unencoded form. To prevent Tomcat rejecting such
> > requests, this attribute may be used to specify the additional characters
> > to allow. If not specified, no addtional characters will be allowed. The
> > value may be any combination of the following characters: " < > [ \ ] ^ ` {
> > | } . Any other characters present in the value will be ignored.
> >
> >
> > As the OPeNDAP spesification (see section 6.1.1.2 on page 22) requires the
> > use of [ and ] in the query string, and several clients doesn't quote these
> > characters, we need to tweak the relaxedQueryChars configuration for tomcat.
>
> For completeness I will add that the DAP protocol also utilizes the < and >
> characters for formulating relational constraints, however these are limited
> to constraining Sequence data types. I don't know that the TDS supports this
> data type so they may not be required in the relaxedQueryChars list .
>
>
> > Regards,
> > Christian Skarby
> >
> > Den tor. 17. jan. 2019 kl. 05:42 skrev Thomas Cook <tmcook@xxxxxxxx>:
> > I do not know how to debug the matlab or python clients, but this is
> > what shows up in the threddsServlet.log
> >
> > Request: "GET
> > /thredds/dodsC/HFR/USWC/6km/hourly/RTV/HFRADAR_US_West_Coast_6km_Resolution_Hourly_RTV_best.ncd.dods?lat,lon,site%5flat,site%5flon,site%5fcode,site%5fnetCode,procParams
> > HTTP/1.0"
> >
> > Thanks
> > Tom
> >
> > On Wed, Jan 16, 2019 at 7:47 PM Dennis Heimbigner <dmh@xxxxxxxx> wrote:
> > >
> > > Is there anyway you can see the exact URL being sent to the server?
> > >
> > > On 1/16/2019 8:24 PM, Thomas Cook wrote:
> > > > Thanks Sean.
> > > > The server is http://hfrnet-tds.ucsd.edu/thredds (although I just
> > > > downgraded back to 4.6.12-SNAPSHOT). I will see about making my test
> > > > server accessible to you, but that will take a little time to
> > > > configure on my end.
> > > > Let me know if you think I should open an issue on the github page and
> > > > move the discussion there.
> > > > Tom
> > > >
> > > > On Wed, Jan 16, 2019 at 6:32 PM Sean Arms <sarms@xxxxxxxx> wrote:
> > > >> Hi Tom,
> > > >>
> > > >> So you are hitting up an opendap endpoint using all three libraries /
> > > >> tools (python, matlab, and panoply)? Is this particular server
> > > >> publicly accessible?
> > > >>
> > > >> I have a bad feeling that this is due to URL encoding (or lack
> > > >> thereof) on the client side and changes made in tomcat 8.5, which it
> > > >> looks like the thredds-docker repository bumped up to with the 4.6.12
> > > >> release.
> > > >>
> > > >> Cheers,
> > > >>
> > > >> Sean
> > > >>
> > > >>
> > > >> On Wed, Jan 16, 2019 at 7:06 PM Thomas Cook <tmcook@xxxxxxxx> wrote:
> > > >>> HI I just wanted to add some details.
> > > >>>
> > > >>> The functionality works when I downgraded to the docker container
> > > >>> 4.6.12-SNAPSHOT
> > > >>>
> > > >>> My data is part of a FMRC collection.
> > > >>>
> > > >>> Thanks,
> > > >>> Tom
> > > >>>
> > > >>> On Wed, Jan 16, 2019 at 4:51 PM Thomas Cook <tmcook@xxxxxxxx> wrote:
> > > >>>> HI
> > > >>>> I just yesterday updated my thredds-docker to 4.6.12 from
> > > >>>> 4.6.12-snapshot. It seems to be running fine, except I'm getting
> > > >>>> errors using matlab ncread and python netcdf4.dataset. When I try to
> > > >>>> access any of the multi-dimensional (time, lat, lon) variables I get
> > > >>>> the following errors
> > > >>>> with matlab:
> > > >>>> Error using netcdflib
> > > >>>> The NetCDF library encountered an error during execution of
> > > >>>> 'getVarsShort' function - 'Access failure (-77)'.
> > > >>>>
> > > >>>> Error in netcdf.getVar (line 136)
> > > >>>> data = netcdflib(funcstr,ncid,varid,varargin{:});
> > > >>>>
> > > >>>> Error in internal.matlab.imagesci.nc/read (line 635)
> > > >>>> data = netcdf.getVar(gid, varid, ...
> > > >>>>
> > > >>>> Error in ncread (line 58)
> > > >>>> vardata = ncObj.read(varName, varargin{:});
> > > >>>>
> > > >>>> Error in readUVfromTDS (line 56)
> > > >>>> u=ncread(url, 'u_mean', [1 1 1], [2 2 2]);
> > > >>>>
> > > >>>>
> > > >>>> and with python:
> > > >>>> netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable.__getitem__()
> > > >>>>
> > > >>>> netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable._get()
> > > >>>>
> > > >>>> netCDF4/_netCDF4.pyx in netCDF4._netCDF4._ensure_nc_success()
> > > >>>>
> > > >>>> RuntimeError: NetCDF: Access failure
> > > >>>>
> > > >>>> I know that the latest docker contains netcdf 4.6, where as the
> > > >>>> snapshot had 4.4. Could it be a version problem with the netCD4
> > > >>>> python
> > > >>>> and matlab libraries? I should be running the latest netcdf python
> > > >>>> version and not too far behind with matlab versions.
> > > >>>>
> > > >>>> The weird things are that the index variables are accessible (time,
> > > >>>> lat, lon etc), and the multiple-dimension data is accessible just
> > > >>>> fine using the OpenDap web interface and Panoply. So I'm a little
> > > >>>> confused. I'm addressing this list since I think there might be some
> > > >>>> matlab and python expertise here and I think there were some messages
> > > >>>> about the Access failure error in the past.
> > > >>>>
> > > >>>> Thanks,
> > > >>>> Tom
> > > >>> _______________________________________________
> > > >>> NOTE: All exchanges posted to Unidata maintained email lists are
> > > >>> recorded in the Unidata inquiry tracking system and made publicly
> > > >>> available through the web. Users who post to any of the lists we
> > > >>> maintain are reminded to remove any personal information that they
> > > >>> do not want to be made public.
> > > >>>
> > > >>>
> > > >>> thredds mailing list
> > > >>> thredds@xxxxxxxxxxxxxxxx
> > > >>> For list information or to unsubscribe, visit:
> > > >>> http://www.unidata.ucar.edu/mailing_lists/
> > > > _______________________________________________
> > > > NOTE: All exchanges posted to Unidata maintained email lists are
> > > > recorded in the Unidata inquiry tracking system and made publicly
> > > > available through the web. Users who post to any of the lists we
> > > > maintain are reminded to remove any personal information that they
> > > > do not want to be made public.
> > > >
> > > >
> > > > thredds mailing list
> > > > thredds@xxxxxxxxxxxxxxxx
> > > > For list information or to unsubscribe, visit:
> > > > http://www.unidata.ucar.edu/mailing_lists/
> > >
> > > _______________________________________________
> > > NOTE: All exchanges posted to Unidata maintained email lists are
> > > recorded in the Unidata inquiry tracking system and made publicly
> > > available through the web. Users who post to any of the lists we
> > > maintain are reminded to remove any personal information that they
> > > do not want to be made public.
> > >
> > >
> > > thredds mailing list
> > > thredds@xxxxxxxxxxxxxxxx
> > > For list information or to unsubscribe, visit:
> > > http://www.unidata.ucar.edu/mailing_lists/
> >
> > _______________________________________________
> > NOTE: All exchanges posted to Unidata maintained email lists are
> > recorded in the Unidata inquiry tracking system and made publicly
> > available through the web. Users who post to any of the lists we
> > maintain are reminded to remove any personal information that they
> > do not want to be made public.
> >
> >
> > thredds mailing list
> > thredds@xxxxxxxxxxxxxxxx
> > For list information or to unsubscribe, visit:
> > http://www.unidata.ucar.edu/mailing_lists/
> > _______________________________________________
> > NOTE: All exchanges posted to Unidata maintained email lists are
> > recorded in the Unidata inquiry tracking system and made publicly
> > available through the web. Users who post to any of the lists we
> > maintain are reminded to remove any personal information that they
> > do not want to be made public.
> >
> >
> > thredds mailing list
> > thredds@xxxxxxxxxxxxxxxx
> > For list information or to unsubscribe, visit:
> > http://www.unidata.ucar.edu/mailing_lists/
>
> = = =
> Nathan Potter ndp at opendap.org
> OPeNDAP, Inc. +1.541.231.3317
>
