Re: [thredds] NetCDF Access failure with matlab and python

  • To: Nathan Potter <ndp@xxxxxxxxxxx>
  • Subject: Re: [thredds] NetCDF Access failure with matlab and python
  • From: Sean Arms <sarms@xxxxxxxx>
  • Date: Fri, 18 Jan 2019 10:59:22 -0700
Let me bring in our Tomcat wrangler, Jen. I thought mod_jk used AJP?

On Fri, Jan 18, 2019 at 10:48 AM Nathan Potter <ndp@xxxxxxxxxxx> wrote:

> Sean,
>
> Interesting.
>
> I thought that mod_jk had been abandoned/superseded for/by the Apache
> JServ Protocol (AJP) protocol with the introduction of Apache 2.x
>
>
> N
>
>
> > On Jan 18, 2019, at 9:21 AM, Sean Arms <sarms@xxxxxxxx> wrote:
> >
> > Hi Nathan, all,
> >
> > netCDF-Java does proper encoding as of 4.6.12 and 5.0.0-beta6 (runtime
> configurable), and in the past it encoded as well - there was a period of
> time in the 4.5 and 4.6 line of development in which the encoding was
> skipped, but that's no longer the case.
> >
> > Interestingly enough, if you run Apache with mod_jk, non-encoded urls
> will be accepted and passed along to tomcat in a usable form without issue,
> but I have no idea how long that will last.
> >
> > Sean
> >
> > On Thu, Jan 17, 2019 at 4:25 PM Nathan Potter <ndp@xxxxxxxxxxx> wrote:
> > Hi Thomas,
> >
> > I believe that I worked with Robert Schmunk (the author of Panoply) on
> this very problem and I think the reason he is using
> netcdfAll-5.0.0-SNAPSHOT is that it solves the outgoing encoding issue on
> the client side. Is that right Robert? (We may not hear back from him as I
> suspect he's furloughed and unavailable a.t.m.)
> >
> > If the client cannot be cajoled into producing the correct URL encoding
> then I think the work falls back to relaxing the server's relaxedQueryChars
> configuration.
> >
> > Which is certainly less than ideal.
> >
> > And to end on a cheery note - once the netcdf-java library is release it
> can take some time (up to a year) before it gets rolled into a Matlab
> release.
> >
> > Sorry I can't be more helpful.
> >
> >
> > Sincerely,
> >
> > Nathan
> >
> >
> > > On Jan 17, 2019, at 1:39 PM, Thomas Cook <tmcook@xxxxxxxx> wrote:
> > >
> > > OK Given Nathan and Christian's comments, I'm focussing my
> > > troubleshooting on client side.
> > > What I've learned
> > > 1) Matlab's native netCDF toolbox depends on netCDF 4.3.3.1. I
> > > couldn't find an easy way to update the netCDF version inside my
> > > Matlab install.
> > > 2) I updated the netcdf4-python module to use netCDF 4.6.12. This did
> > > not solve the issue.
> > > 3) Panoply (which works) is using netcdfAll-5.0.0-SNAPSHOT. I couldn't
> > > find an older version to download, nor was I able to run panoply with
> > > an older netcdfAll.jar
> > > 4) Another user has reported success using the SNTOOLS/MEXNC package
> > > to read the files. This also uses a version of netcdfAll.jar for its
> > > decoding. I can't verify this yet.
> > >
> > > So not sure there's much more I can do with on the client side. I'll
> > > mess with the Dockerfile like Christian suggested, and try using some
> > > non FMRC collections to test with the clients. If anyone has any
> > > ideas, I'm open!
> > > Thanks to all who have replied and I'll update if I make any headway.
> > > Tom
> > >
> > > On Thu, Jan 17, 2019 at 5:53 AM Nathan Potter <ndp@xxxxxxxxxxx> wrote:
> > >>
> > >>
> > >>
> > >>> On Jan 17, 2019, at 5:37 AM, Christian Skarby <christians@xxxxxx>
> wrote:
> > >>>
> > >>> For what it's worth; from what I've seen encoding errors would
> result in a reject from tomcat before the request is passed to the TDS
> application, and it would then probably not show up in the
> threddsServlet.log.
> > >>
> > >> This is the behavior that we have seen.
> > >>
> > >> Tomcat's rejection is an empty document and response headers ala:
> > >>
> > >> HTTP/1.1 400 Bad Request
> > >> Server: Apache-Coyote/1.1
> > >> Transfer-Encoding: chunked
> > >> Date: Thu, 17 Jan 2019 13:47:48 GMT
> > >> Connection: close
> > >>
> > >>>
> > >>> Perhaphs Tom, you could try to just change the application code in
> your docker image (extract war or extracted thredds servlet from one
> version and then inject it in the other). I guess this should be possible,
> although I have not looked into the official docker images for thredds. If
> you make this fly it would be possible to say if it is the servlet or the
> other libraries in the docker image that causes you problems.
> > >>>
> > >>> Regarding the possible encoding errors: Tomcat introduced the more
> strict behavior in v9.0.8 (and also v8.5.31, v8.0.52 and v7.0.87).
> > >>>
> > >>> See the corresponding tomcat changelogs:
> > >>>      •  62273: Implement configuration options to work-around
> specification non-compliant user agents (including all the major browsers)
> that do not correctly %nn encode URI paths and query strings as required by
> RFC 7230 and RFC 3986. (markt)
> > >>> See this thread in the google group for opendap.org:
> > >>>
> https://groups.google.com/a/opendap.org/forum/#!msg/support/ixTqhDXoLZQ/IT0lvZQ7CAAJ
> > >>>
> > >>> Tomcat-documentation:
> > >>> https://tomcat.apache.org/tomcat-9.0-doc/config/http.html
> > >>>
> > >>> relaxedPathChars
> > >>> The HTTP/1.1 specification requires that certain characters are %nn
> encoded when used in URI paths. Unfortunately, many user agents including
> all the major browsers are not compliant with this specification and use
> these characters in unencoded form. To prevent Tomcat rejecting such
> requests, this attribute may be used to specify the additional characters
> to allow. If not specified, no addtional characters will be allowed. The
> value may be any combination of the following characters: " < > [ \ ] ^ ` {
> | } . Any other characters present in the value will be ignored.
> > >>>
> > >>> relaxedQueryChars
> > >>> The HTTP/1.1 specification requires that certain characters are %nn
> encoded when used in URI query strings. Unfortunately, many user agents
> including all the major browsers are not compliant with this specification
> and use these characters in unencoded form. To prevent Tomcat rejecting
> such requests, this attribute may be used to specify the additional
> characters to allow. If not specified, no addtional characters will be
> allowed. The value may be any combination of the following characters: " <
> > [ \ ] ^ ` { | } . Any other characters present in the value will be
> ignored.
> > >>>
> > >>>
> > >>> As the OPeNDAP spesification (see section 6.1.1.2 on page 22)
> requires the use of [ and ] in the query string, and several clients
> doesn't quote these characters, we need to tweak the relaxedQueryChars
> configuration for tomcat.
> > >>
> > >> For completeness I will add that the DAP protocol also utilizes the
> < and > characters for formulating relational constraints, however these
> are limited to constraining Sequence data types. I don't know that the TDS
> supports this data type so they may not be required in the
> relaxedQueryChars list.
> > >>
> > >>
> > >>> Regards,
> > >>> Christian Skarby
> > >>>
> > >>> Den tor. 17. jan. 2019 kl. 05:42 skrev Thomas Cook <tmcook@xxxxxxxx
> >:
> > >>> I do not know how to debug the matlab or python clients, but this is
> > >>> what shows up in the threddsServlet.log
> > >>>
> > >>> Request: "GET
> /thredds/dodsC/HFR/USWC/6km/hourly/RTV/HFRADAR_US_West_Coast_6km_Resolution_Hourly_RTV_best.ncd.dods?lat,lon,site%5flat,site%5flon,site%5fcode,site%5fnetCode,procParams
> > >>> HTTP/1.0"
> > >>>
> > >>> Thanks
> > >>> Tom
> > >>>
> > >>> On Wed, Jan 16, 2019 at 7:47 PM Dennis Heimbigner <dmh@xxxxxxxx>
> wrote:
> > >>>>
> > >>>> Is there anyway you can see the exact URL being sent to the server?
> > >>>>
> > >>>> On 1/16/2019 8:24 PM, Thomas Cook wrote:
> > >>>>> Thanks Sean.
> > >>>>> The server is http://hfrnet-tds.ucsd.edu/thredds (although I just
> > >>>>> downgraded back to 4.6.12-SNAPSHOT). I will see about making my
> test
> > >>>>> server accessible to you, but that will take a little time to
> > >>>>> configure on my end.
> > >>>>> Let me know if you think I should open an issue on the github page
> and
> > >>>>> move the discussion there.
> > >>>>> Tom
> > >>>>>
> > >>>>> On Wed, Jan 16, 2019 at 6:32 PM Sean Arms <sarms@xxxxxxxx> wrote:
> > >>>>>> Hi Tom,
> > >>>>>>
> > >>>>>> So you are hitting up an opendap endpoint using all three
> libraries / tools (python, matlab, and panoply)? Is this particular server
> publicly accessible?
> > >>>>>>
> > >>>>>> I have a bad feeling that this is due to URL encoding (or lack
> thereof) on the client side and changes made in tomcat 8.5, which it looks
> like the thredds-docker repository bumped up to with the 4.6.12 release.
> > >>>>>>
> > >>>>>> Cheers,
> > >>>>>>
> > >>>>>> Sean
> > >>>>>>
> > >>>>>>
> > >>>>>> On Wed, Jan 16, 2019 at 7:06 PM Thomas Cook <tmcook@xxxxxxxx>
> wrote:
> > >>>>>>> HI I just wanted to add some details.
> > >>>>>>>
> > >>>>>>> The functionality works when I downgraded to the docker container
> > >>>>>>> 4.6.12-SNAPSHOT
> > >>>>>>>
> > >>>>>>> My data is part of a FMRC collection.
> > >>>>>>>
> > >>>>>>> Thanks,
> > >>>>>>> Tom
> > >>>>>>>
> > >>>>>>> On Wed, Jan 16, 2019 at 4:51 PM Thomas Cook <tmcook@xxxxxxxx>
> wrote:
> > >>>>>>>> HI
> > >>>>>>>> I just yesterday updated my thredds-docker to 4.6.12 from
> > >>>>>>>> 4.6.12-snapshot. It seems to be running fine, except I'm getting
> > >>>>>>>> errors using matlab ncread and python netcdf4.dataset. When I
> try to
> > >>>>>>>> access any of the multi-dimensional (time, lat, lon) variables
> I get
> > >>>>>>>> the following errors
> > >>>>>>>> with matlab:
> > >>>>>>>> Error using netcdflib
> > >>>>>>>> The NetCDF library encountered an error during execution of
> > >>>>>>>> 'getVarsShort' function - 'Access failure (-77)'.
> > >>>>>>>>
> > >>>>>>>> Error in netcdf.getVar (line 136)
> > >>>>>>>>     data = netcdflib(funcstr,ncid,varid,varargin{:});
> > >>>>>>>>
> > >>>>>>>> Error in internal.matlab.imagesci.nc/read (line 635)
> > >>>>>>>>                 data  = netcdf.getVar(gid, varid, ...
> > >>>>>>>>
> > >>>>>>>> Error in ncread (line 58)
> > >>>>>>>> vardata = ncObj.read(varName, varargin{:});
> > >>>>>>>>
> > >>>>>>>> Error in readUVfromTDS (line 56)
> > >>>>>>>> u=ncread(url, 'u_mean', [1 1 1], [2 2 2]);
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> and with python:
> > >>>>>>>> netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable.__getitem__()
> > >>>>>>>>
> > >>>>>>>> netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable._get()
> > >>>>>>>>
> > >>>>>>>> netCDF4/_netCDF4.pyx in netCDF4._netCDF4._ensure_nc_success()
> > >>>>>>>>
> > >>>>>>>> RuntimeError: NetCDF: Access failure
> > >>>>>>>>
> > >>>>>>>> I know that the latest docker contains netcdf 4.6, where as the
> > >>>>>>>> snapshot had 4.4. Could it be a version problem with the netCD4
> python
> > >>>>>>>> and matlab libraries? I should be running the latest netcdf
> python
> > >>>>>>>> version and not too far behind with matlab versions.
> > >>>>>>>>
> > >>>>>>>> The weird things are that the index variables are accessible
> (time,
> > >>>>>>>> lat, lon etc),  and the multiple-dimension data is accessible
> just
> > >>>>>>>> fine using the OpenDap web interface and Panoply. So I'm a
> little
> > >>>>>>>> confused. I'm addressing this list since I think there might be
> some
> > >>>>>>>> matlab and python expertise here and I think there were some
> messages
> > >>>>>>>> about the Access failure error in the past.
> > >>>>>>>>
> > >>>>>>>> Thanks,
> > >>>>>>>> Tom
> > >>>>>>> _______________________________________________
> > >>>>>>> NOTE: All exchanges posted to Unidata maintained email lists are
> > >>>>>>> recorded in the Unidata inquiry tracking system and made publicly
> > >>>>>>> available through the web.  Users who post to any of the lists we
> > >>>>>>> maintain are reminded to remove any personal information that
> they
> > >>>>>>> do not want to be made public.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> thredds mailing list
> > >>>>>>> thredds@xxxxxxxxxxxxxxxx
> > >>>>>>> For list information or to unsubscribe,  visit:
> http://www.unidata.ucar.edu/mailing_lists/
> > >>>>> _______________________________________________
> > >>>>> NOTE: All exchanges posted to Unidata maintained email lists are
> > >>>>> recorded in the Unidata inquiry tracking system and made publicly
> > >>>>> available through the web.  Users who post to any of the lists we
> > >>>>> maintain are reminded to remove any personal information that they
> > >>>>> do not want to be made public.
> > >>>>>
> > >>>>>
> > >>>>> thredds mailing list
> > >>>>> thredds@xxxxxxxxxxxxxxxx
> > >>>>> For list information or to unsubscribe,  visit:
> http://www.unidata.ucar.edu/mailing_lists/
> > >>>>
> > >>>> _______________________________________________
> > >>>> NOTE: All exchanges posted to Unidata maintained email lists are
> > >>>> recorded in the Unidata inquiry tracking system and made publicly
> > >>>> available through the web.  Users who post to any of the lists we
> > >>>> maintain are reminded to remove any personal information that they
> > >>>> do not want to be made public.
> > >>>>
> > >>>>
> > >>>> thredds mailing list
> > >>>> thredds@xxxxxxxxxxxxxxxx
> > >>>> For list information or to unsubscribe,  visit:
> http://www.unidata.ucar.edu/mailing_lists/
> > >>>
> > >>> _______________________________________________
> > >>> NOTE: All exchanges posted to Unidata maintained email lists are
> > >>> recorded in the Unidata inquiry tracking system and made publicly
> > >>> available through the web.  Users who post to any of the lists we
> > >>> maintain are reminded to remove any personal information that they
> > >>> do not want to be made public.
> > >>>
> > >>>
> > >>> thredds mailing list
> > >>> thredds@xxxxxxxxxxxxxxxx
> > >>> For list information or to unsubscribe,  visit:
> http://www.unidata.ucar.edu/mailing_lists/
> > >>> _______________________________________________
> > >>> NOTE: All exchanges posted to Unidata maintained email lists are
> > >>> recorded in the Unidata inquiry tracking system and made publicly
> > >>> available through the web.  Users who post to any of the lists we
> > >>> maintain are reminded to remove any personal information that they
> > >>> do not want to be made public.
> > >>>
> > >>>
> > >>> thredds mailing list
> > >>> thredds@xxxxxxxxxxxxxxxx
> > >>> For list information or to unsubscribe,  visit:
> http://www.unidata.ucar.edu/mailing_lists/
> > >>
> > >> = = =
> > >> Nathan Potter                        ndp at opendap.org
> > >> OPeNDAP, Inc.                        +1.541.231.3317
> > >>
> >
> > = = =
> > Nathan Potter                        ndp at opendap.org
> > OPeNDAP, Inc.                        +1.541.231.3317
> >
> > _______________________________________________
> > NOTE: All exchanges posted to Unidata maintained email lists are
> > recorded in the Unidata inquiry tracking system and made publicly
> > available through the web.  Users who post to any of the lists we
> > maintain are reminded to remove any personal information that they
> > do not want to be made public.
> >
> >
> > thredds mailing list
> > thredds@xxxxxxxxxxxxxxxx
> > For list information or to unsubscribe,  visit:
> http://www.unidata.ucar.edu/mailing_lists/
>
> = = =
> Nathan Potter                        ndp at opendap.org
> OPeNDAP, Inc.                        +1.541.231.3317
>
>
  • 2019 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: