Re: [thredds] TDS5 Beta 2 and latest version of Tomcat

  • To: Roy Mendelssohn - NOAA Federal <roy.mendelssohn@xxxxxxxx>
  • Subject: Re: [thredds] TDS5 Beta 2 and latest version of Tomcat
  • From: Christian Skarby <christians@xxxxxx>
  • Date: Tue, 31 Jul 2018 08:29:49 +0200
Indeed this is a change in tomcat behavior since v9.0.8 (and also v8.5.31,
v8.0.52 and v7.0.87). I don't think it's possible to do a workaround in the
application code; from what I've seen tomcat rejects requests with [ and ]
before the request is handed over to the application code. (I can confirm
that this is also affecting thredds v4.6.11.)

See the corresponding tomcat changelogs:

   - [image: Add:] 62273
   <http://bz.apache.org/bugzilla/show_bug.cgi?id=62273>: Implement
   configuration options to work-around specification non-compliant user
   agents (including all the major browsers) that do not correctly %nn encode
   URI paths and query strings as required by RFC 7230 and RFC 3986. (markt)

See this thread in the google group for opendap.org:
https://groups.google.com/a/opendap.org/forum/#!msg/support/ixTqhDXoLZQ/
IT0lvZQ7CAAJ

Tomcat-documentation:
https://tomcat.apache.org/tomcat-9.0-doc/config/http.html

relaxedPathChars

The HTTP/1.1 specification <https://tools.ietf.org/rfc/rfc7230.txt> requires
that certain characters are %nn encoded when used in URI paths.
Unfortunately, many user agents including all the major browsers are not
compliant with this specification and use these characters in unencoded
form. To prevent Tomcat rejecting such requests, this attribute may be used
to specify the additional characters to allow. If not specified, no
addtional characters will be allowed. The value may be any combination of
the following characters: " < > [ \ ] ^ ` { | } . Any other characters
present in the value will be ignored.
relaxedQueryChars

The HTTP/1.1 specification <https://tools.ietf.org/rfc/rfc7230.txt> requires
that certain characters are %nn encoded when used in URI query strings.
Unfortunately, many user agents including all the major browsers are not
compliant with this specification and use these characters in unencoded
form. To prevent Tomcat rejecting such requests, this attribute may be used
to specify the additional characters to allow. If not specified, no
addtional characters will be allowed. The value may be any combination of
the following characters: " < > [ \ ] ^ ` { | } . Any other characters
present in the value will be ignored.

As the OPeNDAP spesification (see section 6.1.1.2 on page 22)
<https://cdn.earthdata.nasa.gov/conduit/upload/512/ESE-RFC-004v1.1.pdf>
requires
the use of [ and ] in the query string, and several clients doesn't quote
these characters, we need to tweak the relaxedQueryChars configuration for
tomcat.

--
Best regards,

Christian Skarby
MET Norway, IT

2018-07-31 1:31 GMT+02:00 Roy Mendelssohn - NOAA Federal <
roy.mendelssohn@xxxxxxxx>:

> BTW - I doubt that this is unique to TDS 5.0Beta2,  but probably is true
> for TDS4.6.xx also.  I just happened to be testing TDS 5.0Beta2 to see what
> if any changes I needed to make to our catalogs.
>
> -Roy
>
>
> > On Jul 30, 2018, at 4:12 PM, Roy Mendelssohn - NOAA Federal <
> roy.mendelssohn@xxxxxxxx> wrote:
> >
> > Okay adding the relaxedQueryChars="[]" to server.xml did indeed appear
> to fix the problem,  not only in TDS but in other apps.  But is this the
> best place to fix it rather than in the app,  and even more before I switch
> over I would want to do more testing that it fixes things everywhere.  So
> clearly something did change in Tomcat,  because I was pretty certain it
> was the square brackets that were causing the problem  (anything that
> didn't have a square bracket worked just fine).  I also need to do some
> testing if this fixes things with applications that make queries to the
> TDS, rather than URL's generated by the TDS.
> >
> > This is not the first time that changes in Tomcat like this broke code.
> It was maybe 1.5 - 2 years ago they made some changes in how they dealt
> with percent encoding that also affected a lot of things.
> >
> > Thanks,
> >
> > -Roy
> >
> >
> >> On Jul 30, 2018, at 3:42 PM, Dennis Heimbigner <dmh@xxxxxxxx> wrote:
> >>
> >> Seeing this twice makes me think that indeed tomcat has changed its
> behavior.
> >> Isn't the saying "twice is enemy action..."?
> >> The alternative, I think,  is to change the opendap .html response to
> use the % encoded form
> >> for '[' and ']' when it constructs requests. WIll ncss have same
> problem?
> >> In any case,I do not know if changing the .html response would cause
> trouble when using
> >> older versions of tomcat that by default allow [].
> >> =Dennis Heimbigner
> >>  Unidata
> >>
> >> On 7/30/2018 4:36 PM, Sean Arms wrote:
> >>> Greetings Roy!
> >>>
> >>> Thank you for your report! We had a user contact us last week thorough
> our support channel who was having an issue with OPeNDAP requests not
> working, due to [ and ], much like you report. This user found the
> following tweak to Tomcat's server.xml that fixed the problem
> (specifically, the relaxedQueryChars attribute):
> >>>
> >>> <Connector port="8080" protocol="HTTP/1.1"
> >>> connectionTimeout="20000"
> >>> redirectPort="8443" relaxedQueryChars="[]" />
> >>>
> >>> The initial thought was that this was something particular to their
> server, but I think your correct in that it's a change in how tomcat
> handles these characters. The user who emailed us didn't include the
> version of Tomcat they were using, so I'm not 100% sure, but this sure
> sounds like the issue.
> >>>
> >>> What I don't know yet is if this must be configured at a level above
> the TDS (like tomcat's server.xml), or if this something we could address
> in the TDS codebase (in a servlet container agnostic way).
> >>>
> >>> In your case, if you try the request but encode the square brackets by
> hand using %5B for '[' and %5D for ']', does the query work?
> >>>
> >>> Thank you again!
> >>>
> >>> Sean
> >>>
> >>>
> >>>
> >>>
> >>> On Mon, Jul 30, 2018 at 12:58 PM Roy Mendelssohn - NOAA Federal <
> roy.mendelssohn@xxxxxxxx<mailto:roy.mendelssohn@xxxxxxxx>> wrote:
> >>>
> >>>   Hi All:
> >>>
> >>>   Something we discovered is that Tomcat has made some changes in
> >>>   how it deals with percent encoding, and it appears to not work
> >>>   with the TDS5 Beta2 release.  If I run in Tomcat 8.5.14, using the
> >>>   default TDS settings that are installed if nothing else is there,
> >>>   if I go to:
> >>>
> >>>   http://localhost:8080/thredds/dodsC/testAll/2004050312_eta_
> 211.nc.html
> >>>
> >>>   select datetime and asked for ASCII,  it works.
> >>>
> >>>   If I do the exact same thing with Tomcat 8.5.32,  I get a blank
> >>>   page.  From looking at the Tomcat changelogs,  I think there may
> >>>   have been a change at 8.5.15. It appears to be something to do
> >>>   with  the handling of '[' and ']'.
> >>>
> >>>   HTH,
> >>>
> >>>   -Roy
> >>>
> >>>
> >>>
> >>>   **********************
> >>>   "The contents of this message do not reflect any position of the
> >>>   U.S. Government or NOAA."
> >>>   **********************
> >>>   Roy Mendelssohn
> >>>   Supervisory Operations Research Analyst
> >>>   NOAA/NMFS
> >>>   Environmental Research Division
> >>>   Southwest Fisheries Science Center
> >>>   ***Note new street address***
> >>>   110 McAllister Way
> >>>   Santa Cruz, CA 95060
> >>>   Phone: (831)-420-3666
> >>>   Fax: (831) 420-3980
> >>>   e-mail: Roy.Mendelssohn@xxxxxxxx <mailto:Roy.Mendelssohn@xxxxxxxx>
> >>>   www: http://www.pfeg.noaa.gov/
> >>>
> >>>   "Old age and treachery will overcome youth and skill."
> >>>   "From those who have been given much, much will be expected"
> >>>   "the arc of the moral universe is long, but it bends toward
> >>>   justice" -MLK Jr.
> >>>
> >>>   _______________________________________________
> >>>   NOTE: All exchanges posted to Unidata maintained email lists are
> >>>   recorded in the Unidata inquiry tracking system and made publicly
> >>>   available through the web.  Users who post to any of the lists we
> >>>   maintain are reminded to remove any personal information that they
> >>>   do not want to be made public.
> >>>
> >>>
> >>>   thredds mailing list
> >>>   thredds@xxxxxxxxxxxxxxxx <mailto:thredds@xxxxxxxxxxxxxxxx>
> >>>   For list information or to unsubscribe,  visit:
> >>>   http://www.unidata.ucar.edu/mailing_lists/
> >>>
> >>>
> >>>
> >>> _______________________________________________
> >>> NOTE: All exchanges posted to Unidata maintained email lists are
> >>> recorded in the Unidata inquiry tracking system and made publicly
> >>> available through the web.  Users who post to any of the lists we
> >>> maintain are reminded to remove any personal information that they
> >>> do not want to be made public.
> >>>
> >>>
> >>> thredds mailing list
> >>> thredds@xxxxxxxxxxxxxxxx
> >>> For list information or to unsubscribe,  visit:
> http://www.unidata.ucar.edu/mailing_lists/
> >
> > **********************
> > "The contents of this message do not reflect any position of the U.S.
> Government or NOAA."
> > **********************
> > Roy Mendelssohn
> > Supervisory Operations Research Analyst
> > NOAA/NMFS
> > Environmental Research Division
> > Southwest Fisheries Science Center
> > ***Note new street address***
> > 110 McAllister Way
> > Santa Cruz, CA 95060
> > Phone: (831)-420-3666
> > Fax: (831) 420-3980
> > e-mail: Roy.Mendelssohn@xxxxxxxx www: http://www.pfeg.noaa.gov/
> >
> > "Old age and treachery will overcome youth and skill."
> > "From those who have been given much, much will be expected"
> > "the arc of the moral universe is long, but it bends toward justice"
> -MLK Jr.
> >
>
> **********************
> "The contents of this message do not reflect any position of the U.S.
> Government or NOAA."
> **********************
> Roy Mendelssohn
> Supervisory Operations Research Analyst
> NOAA/NMFS
> Environmental Research Division
> Southwest Fisheries Science Center
> ***Note new street address***
> 110 McAllister Way
> Santa Cruz, CA 95060
> Phone: (831)-420-3666
> Fax: (831) 420-3980
> e-mail: Roy.Mendelssohn@xxxxxxxx www: http://www.pfeg.noaa.gov/
>
> "Old age and treachery will overcome youth and skill."
> "From those who have been given much, much will be expected"
> "the arc of the moral universe is long, but it bends toward justice" -MLK
> Jr.
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web.  Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> thredds mailing list
> thredds@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe,  visit:
> http://www.unidata.ucar.edu/mailing_lists/
>
  • 2018 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: