Hi Thomas,
I believe that I worked with Robert Schmunk (the author of Panoply) on this
very problem and I think the reason he is using netcdfAll-5.0.0-SNAPSHOT is
that it solves the outgoing encoding issue on the client side. Is that right
Robert? (We may not hear back from him as I suspect he's furloughed and
unavailable a.t.m.)
If the client cannot be cajoled into producing the correct URL encoding then I
think the work falls back to relaxing the server's relaxedQueryChars
configuration.
Which is certainly less than ideal.
And to end on a cheery note - once the netcdf-java library is release it can
take some time (up to a year) before it gets rolled into a Matlab release.
Sorry I can't be more helpful.
Sincerely,
Nathan
> On Jan 17, 2019, at 1:39 PM, Thomas Cook <tmcook@xxxxxxxx> wrote:
>
> OK Given Nathan and Christian's comments, I'm focussing my
> troubleshooting on client side.
> What I've learned
> 1) Matlab's native netCDF toolbox depends on netCDF 4.3.3.1. I
> couldn't find an easy way to update the netCDF version inside my
> Matlab install.
> 2) I updated the netcdf4-python module to use netCDF 4.6.12. This did
> not solve the issue.
> 3) Panoply (which works) is using netcdfAll-5.0.0-SNAPSHOT. I couldn't
> find an older version to download, nor was I able to run panoply with
> an older netcdfAll.jar
> 4) Another user has reported success using the SNTOOLS/MEXNC package
> to read the files. This also uses a version of netcdfAll.jar for its
> decoding. I can't verify this yet.
>
> So not sure there's much more I can do with on the client side. I'll
> mess with the Dockerfile like Christian suggested, and try using some
> non FMRC collections to test with the clients. If anyone has any
> ideas, I'm open!
> Thanks to all who have replied and I'll update if I make any headway.
> Tom
>
> On Thu, Jan 17, 2019 at 5:53 AM Nathan Potter <ndp@xxxxxxxxxxx> wrote:
>>
>>
>>
>>> On Jan 17, 2019, at 5:37 AM, Christian Skarby <christians@xxxxxx> wrote:
>>>
>>> For what it's worth; from what I've seen encoding errors would result in a
>>> reject from tomcat before the request is passed to the TDS application, and
>>> it would then probably not show up in the threddsServlet.log.
>>
>> This is the behavior that we have seen.
>>
>> Tomcat's rejection is an empty document and response headers ala:
>>
>> HTTP/1.1 400 Bad Request
>> Server: Apache-Coyote/1.1
>> Transfer-Encoding: chunked
>> Date: Thu, 17 Jan 2019 13:47:48 GMT
>> Connection: close
>>
>>>
>>> Perhaphs Tom, you could try to just change the application code in your
>>> docker image (extract war or extracted thredds servlet from one version and
>>> then inject it in the other). I guess this should be possible, although I
>>> have not looked into the official docker images for thredds. If you make
>>> this fly it would be possible to say if it is the servlet or the other
>>> libraries in the docker image that causes you problems.
>>>
>>> Regarding the possible encoding errors: Tomcat introduced the more strict
>>> behavior in v9.0.8 (and also v8.5.31, v8.0.52 and v7.0.87).
>>>
>>> See the corresponding tomcat changelogs:
>>> • 62273: Implement configuration options to work-around specification
>>> non-compliant user agents (including all the major browsers) that do not
>>> correctly %nn encode URI paths and query strings as required by RFC 7230
>>> and RFC 3986. (markt)
>>> See this thread in the google group for opendap.org:
>>> https://groups.google.com/a/opendap.org/forum/#!msg/support/ixTqhDXoLZQ/IT0lvZQ7CAAJ
>>>
>>> Tomcat-documentation:
>>> https://tomcat.apache.org/tomcat-9.0-doc/config/http.html
>>>
>>> relaxedPathChars
>>> The HTTP/1.1 specification requires that certain characters are %nn encoded
>>> when used in URI paths. Unfortunately, many user agents including all the
>>> major browsers are not compliant with this specification and use these
>>> characters in unencoded form. To prevent Tomcat rejecting such requests,
>>> this attribute may be used to specify the additional characters to allow.
>>> If not specified, no addtional characters will be allowed. The value may be
>>> any combination of the following characters: " < > [ \ ] ^ ` { | } . Any
>>> other characters present in the value will be ignored.
>>>
>>> relaxedQueryChars
>>> The HTTP/1.1 specification requires that certain characters are %nn encoded
>>> when used in URI query strings. Unfortunately, many user agents including
>>> all the major browsers are not compliant with this specification and use
>>> these characters in unencoded form. To prevent Tomcat rejecting such
>>> requests, this attribute may be used to specify the additional characters
>>> to allow. If not specified, no addtional characters will be allowed. The
>>> value may be any combination of the following characters: " < > [ \ ] ^ ` {
>>> | } . Any other characters present in the value will be ignored.
>>>
>>>
>>> As the OPeNDAP spesification (see section 6.1.1.2 on page 22) requires the
>>> use of [ and ] in the query string, and several clients doesn't quote these
>>> characters, we need to tweak the relaxedQueryChars configuration for tomcat.
>>
>> For completeness I will add that the DAP protocol also utilizes the < and >
>> characters for formulating relational constraints, however these are limited
>> to constraining Sequence data types. I don't know that the TDS supports this
>> data type so they may not be required in the relaxedQueryChars list.
>>
>>
>>> Regards,
>>> Christian Skarby
>>>
>>> Den tor. 17. jan. 2019 kl. 05:42 skrev Thomas Cook <tmcook@xxxxxxxx>:
>>> I do not know how to debug the matlab or python clients, but this is
>>> what shows up in the threddsServlet.log
>>>
>>> Request: "GET
>>> /thredds/dodsC/HFR/USWC/6km/hourly/RTV/HFRADAR_US_West_Coast_6km_Resolution_Hourly_RTV_best.ncd.dods?lat,lon,site%5flat,site%5flon,site%5fcode,site%5fnetCode,procParams
>>> HTTP/1.0"
>>>
>>> Thanks
>>> Tom
>>>
>>> On Wed, Jan 16, 2019 at 7:47 PM Dennis Heimbigner <dmh@xxxxxxxx> wrote:
>>>>
>>>> Is there anyway you can see the exact URL being sent to the server?
>>>>
>>>> On 1/16/2019 8:24 PM, Thomas Cook wrote:
>>>>> Thanks Sean.
>>>>> The server is http://hfrnet-tds.ucsd.edu/thredds (although I just
>>>>> downgraded back to 4.6.12-SNAPSHOT). I will see about making my test
>>>>> server accessible to you, but that will take a little time to
>>>>> configure on my end.
>>>>> Let me know if you think I should open an issue on the github page and
>>>>> move the discussion there.
>>>>> Tom
>>>>>
>>>>> On Wed, Jan 16, 2019 at 6:32 PM Sean Arms <sarms@xxxxxxxx> wrote:
>>>>>> Hi Tom,
>>>>>>
>>>>>> So you are hitting up an opendap endpoint using all three libraries /
>>>>>> tools (python, matlab, and panoply)? Is this particular server publicly
>>>>>> accessible?
>>>>>>
>>>>>> I have a bad feeling that this is due to URL encoding (or lack thereof)
>>>>>> on the client side and changes made in tomcat 8.5, which it looks like
>>>>>> the thredds-docker repository bumped up to with the 4.6.12 release.
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Sean
>>>>>>
>>>>>>
>>>>>> On Wed, Jan 16, 2019 at 7:06 PM Thomas Cook <tmcook@xxxxxxxx> wrote:
>>>>>>> HI I just wanted to add some details.
>>>>>>>
>>>>>>> The functionality works when I downgraded to the docker container
>>>>>>> 4.6.12-SNAPSHOT
>>>>>>>
>>>>>>> My data is part of a FMRC collection.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Tom
>>>>>>>
>>>>>>> On Wed, Jan 16, 2019 at 4:51 PM Thomas Cook <tmcook@xxxxxxxx> wrote:
>>>>>>>> HI
>>>>>>>> I just yesterday updated my thredds-docker to 4.6.12 from
>>>>>>>> 4.6.12-snapshot. It seems to be running fine, except I'm getting
>>>>>>>> errors using matlab ncread and python netcdf4.dataset. When I try to
>>>>>>>> access any of the multi-dimensional (time, lat, lon) variables I get
>>>>>>>> the following errors
>>>>>>>> with matlab:
>>>>>>>> Error using netcdflib
>>>>>>>> The NetCDF library encountered an error during execution of
>>>>>>>> 'getVarsShort' function - 'Access failure (-77)'.
>>>>>>>>
>>>>>>>> Error in netcdf.getVar (line 136)
>>>>>>>> data = netcdflib(funcstr,ncid,varid,varargin{:});
>>>>>>>>
>>>>>>>> Error in internal.matlab.imagesci.nc/read (line 635)
>>>>>>>> data = netcdf.getVar(gid, varid, ...
>>>>>>>>
>>>>>>>> Error in ncread (line 58)
>>>>>>>> vardata = ncObj.read(varName, varargin{:});
>>>>>>>>
>>>>>>>> Error in readUVfromTDS (line 56)
>>>>>>>> u=ncread(url, 'u_mean', [1 1 1], [2 2 2]);
>>>>>>>>
>>>>>>>>
>>>>>>>> and with python:
>>>>>>>> netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable.__getitem__()
>>>>>>>>
>>>>>>>> netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable._get()
>>>>>>>>
>>>>>>>> netCDF4/_netCDF4.pyx in netCDF4._netCDF4._ensure_nc_success()
>>>>>>>>
>>>>>>>> RuntimeError: NetCDF: Access failure
>>>>>>>>
>>>>>>>> I know that the latest docker contains netcdf 4.6, where as the
>>>>>>>> snapshot had 4.4. Could it be a version problem with the netCD4 python
>>>>>>>> and matlab libraries? I should be running the latest netcdf python
>>>>>>>> version and not too far behind with matlab versions.
>>>>>>>>
>>>>>>>> The weird things are that the index variables are accessible (time,
>>>>>>>> lat, lon etc), and the multiple-dimension data is accessible just
>>>>>>>> fine using the OpenDap web interface and Panoply. So I'm a little
>>>>>>>> confused. I'm addressing this list since I think there might be some
>>>>>>>> matlab and python expertise here and I think there were some messages
>>>>>>>> about the Access failure error in the past.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Tom
>>>>>>> _______________________________________________
>>>>>>> NOTE: All exchanges posted to Unidata maintained email lists are
>>>>>>> recorded in the Unidata inquiry tracking system and made publicly
>>>>>>> available through the web. Users who post to any of the lists we
>>>>>>> maintain are reminded to remove any personal information that they
>>>>>>> do not want to be made public.
>>>>>>>
>>>>>>>
>>>>>>> thredds mailing list
>>>>>>> thredds@xxxxxxxxxxxxxxxx
>>>>>>> For list information or to unsubscribe, visit:
>>>>>>> http://www.unidata.ucar.edu/mailing_lists/
>>>>> _______________________________________________
>>>>> NOTE: All exchanges posted to Unidata maintained email lists are
>>>>> recorded in the Unidata inquiry tracking system and made publicly
>>>>> available through the web. Users who post to any of the lists we
>>>>> maintain are reminded to remove any personal information that they
>>>>> do not want to be made public.
>>>>>
>>>>>
>>>>> thredds mailing list
>>>>> thredds@xxxxxxxxxxxxxxxx
>>>>> For list information or to unsubscribe, visit:
>>>>> http://www.unidata.ucar.edu/mailing_lists/
>>>>
>>>> _______________________________________________
>>>> NOTE: All exchanges posted to Unidata maintained email lists are
>>>> recorded in the Unidata inquiry tracking system and made publicly
>>>> available through the web. Users who post to any of the lists we
>>>> maintain are reminded to remove any personal information that they
>>>> do not want to be made public.
>>>>
>>>>
>>>> thredds mailing list
>>>> thredds@xxxxxxxxxxxxxxxx
>>>> For list information or to unsubscribe, visit:
>>>> http://www.unidata.ucar.edu/mailing_lists/
>>>
>>> _______________________________________________
>>> NOTE: All exchanges posted to Unidata maintained email lists are
>>> recorded in the Unidata inquiry tracking system and made publicly
>>> available through the web. Users who post to any of the lists we
>>> maintain are reminded to remove any personal information that they
>>> do not want to be made public.
>>>
>>>
>>> thredds mailing list
>>> thredds@xxxxxxxxxxxxxxxx
>>> For list information or to unsubscribe, visit:
>>> http://www.unidata.ucar.edu/mailing_lists/
>>> _______________________________________________
>>> NOTE: All exchanges posted to Unidata maintained email lists are
>>> recorded in the Unidata inquiry tracking system and made publicly
>>> available through the web. Users who post to any of the lists we
>>> maintain are reminded to remove any personal information that they
>>> do not want to be made public.
>>>
>>>
>>> thredds mailing list
>>> thredds@xxxxxxxxxxxxxxxx
>>> For list information or to unsubscribe, visit:
>>> http://www.unidata.ucar.edu/mailing_lists/
>>
>> = = =
>> Nathan Potter ndp at opendap.org
>> OPeNDAP, Inc. +1.541.231.3317
>>
= = =
Nathan Potter ndp at opendap.org
OPeNDAP, Inc. +1.541.231.3317
