HTTP Session Control

Dennis Heimbigner
UCAR/Unidata

Date: July 20, 2011
Revised: September 25, 2011

Introduction

New public classes are introduced to the package ucar.nc2.util.net. These are HTTPSession.java, HTTPMethod.java, HTTPMethodStream.java, HTTPException.java, These classes are intended to wrap various Apache httpclient-3 library classes so that (1) it will simplify use, and (2) it will be possible to later switch to, say, the Apache httpclient4 library.

These classes support a form of session semantics for Java network access. The notion of session is only loosely tied to the HTTP notion of session.

HTTPSession and Session Semantics

A session is encapsulated in an instance of the class HTTPSession. The encapsulation is with respect to a specific url. This means that once a session is specified, it is tied permanently to that url and "compatible" urls.

As a special case for backward compatibility, an HTTPSession constructor with no arguments is provided. If used, then (1) the session is tied to no specific url, and (2) method creation must specify a url (see HTTPMethod).

It is important to note that Session objects do NOT correspond with the HttpClient objects of the Apache httpclient library. The Session described here encapulates an instance of an Apache HttpClient, but Sessions also wrap and control httpclient library methods such as GetMethod via the class HTTPMethod. This is so it can ensure that the Session - url correspondence is not violated (see here for additional details).

HTTPMethod

This class encapsulates the information about a given method request and response. Its primary operation is execute(), which causes a request to be sent to a server and a response obtained.

The HTTPMethod operates as a factory for creating HTTPMethod instances. To create a method for doing an HTTP GET operation, for example, one would invoke this operation

HTTPMethod m = HTTPMethod.Get(session);

where "session" is a previously created session object.

Alternately, one can create a method and specify a url.

HTTPMethod m = HTTPMethod.Get(session,url);

See the section on specifying a url when creating an HTTPMethod instance for more information.

HTTPMethodStream

The purpose of this class is to allow other classes to access the data stream associated with a method response. It tracks the method and the session to allow them to be closed when the stream hits eof.

HTTPException

This class is a subclass of java.io.IOException. It is the exception for reporting errors out of the classes listed above.

Examples

Example 1: Create/Use/Release Cycle

public class Main
{
    public static void main(String[] argv)
    {
	String url = argv[0];
        HTTPSession session = new HTTPSession(url);
        HTTPMethod method = HTTPMethod.Get(session);
        int status = method.execute();
        System.out.printf("Execute: status code = %d\n", status);
	method.close();
	session.close();
    }
}

Example 2: Quick Request/Response Cycle

public class Main
{
    public static void main(String[] argv)
    {
	String url = argv[0[];
	// The class HttpClientManager has a number of static
        // methods that wrap HTTPSession and HTTPMethod.
	// Specifyin null as the first argument forces HttpClientManager
        // to create and destroy an internal HTTPSession instance.
	string content = HttpClientManager.getUrlContentsAsString(null,url,1024);
    }
}

Example 3: Setting Some Global Parameters

public class Main
{
    public static void main(String[] argv)
    {
	String url = argv[0];

        HTTPSession.setGlobalCredentialsProvider(new UserPasswordProvider());
	HTTPSession.setGlobalUserAgent("netcdf/java");
	HTTPSession.setMaxConnections(4);
	HTTPSession.setGlobalAuthenticationPreemptive(true);

        HTTPSession session = new HTTPSession(url);
        HTTPMethod method = HTTPMethod.Get(session);
        int status = method.execute();
        System.out.printf("Execute: status code = %d\n", status);
	method.close();
	session.close();
    }
}

Example 4: Setting Some Local Parameters

public class Main
{
    public static void main(String[] argv)
    {
	String url = argv[0];

        HTTPSession session = new HTTPSession(url);
        session.setCredentialsProvider(new UserPasswordProvider());
	session.setAuthenticationPreemptive(true);
	session.setUserAgent("agent");
	session.setConnectionManagerTimeout(475);
	session.setSoTimeout(475);

        HTTPMethod method = HTTPMethod.Get(session);
        int status = method.execute();
        System.out.printf("Execute: status code = %d\n", status);
	method.close();
	session.close();
    }
}

Specifying a url when creating an HTTPMethod Instance

It is possible to specify a url when invoking, for example, HTTPMethod.Get. This is because the url argument to the HTTPSession constructor actually serves two purposes. First, if the method is created without specifying a url, then the session url is used to specify the data to be retrieved by the method invocation.

Second, if the method is created and specifies a url, for example,

HTTPMethod m = HTTPMethod.Get(session,url2);

this second url is used to specify the data to be retrieved by the method invocation.

This might occur (and does) if, for example, the url given to HTTPSession represented some general url such as http://motherlode.ucar.edu/path/file.nc and the url given to HTTPMethod.Get was for something more specific such as http://motherlode.ucar.edu/path/file.nc.dds.

The important point is that this second url must be "compatible" with the session url. The term "compatible" basically means that the HTTPSession url, as a string, must be a prefix of the url given to HTTPMethod.Get. This maintains the semantics of the Session but allows flexibility in accessing data from the server.

There are some exceptions to the prefix rule for compatibility. Let surl be the session url and murl be the url specified for the method. The following rule holds.

• If the surl does not have a principal, then the murl without its principal is tested for compatibility. In effect, a missing principal acts like a wildcard.

  Similarly, missing ports or paths in the surl also act like wildcards.

Additionally, and for backward compatibility, if surl is null or is the empty string (""), then murl must be specified and any such url is considered compatible.

Authorization Credentials

HTTPSession operates in conjunction with the new credentialing mechanisms to support better mechanisms for setting authorization credentials.

The key idea is that a single, global database of credentials is maintained. The key for the database is the combination of the authorization scheme plus a url. This key pair maps to an instance of CredentialsProvider. At the time an HTTP method is executed, the url indicates when to apply authorization (if the server requests it). The scheme indicates the kind of authorization scheme is being used: HTTP Basic or Digest for example. The credentials provider is then invoked to compute the set of credentials to be sent to the server.

Currently the following schemes are supported.

Basic: the HTTP Basic scheme based on clear-text user name and password.

Digest: the HTTP digest scheme based on encrypted user name and password.

Proxy: a scheme for accessing a proxy server based on clear-text user name and password. Note that there is an alternate simple proxy" that does not involve authentication, but only requires a host name and a port number.

SSL: a scheme that uses a client-side key to authenticate the client to the server. The ssl scheme is usually part of an SSL connection where the server authenticates to the client and then the client authenticates to the server.

The credentials provider (see HTTPSSLProvider) is used in a non-standard way and it contains the following information.

• The path to the keystore file
• The password for decrypting the keystore file.
• The path to a truststore file
• The password for decrypting the truststore file.

The last two items are optional. If missing, then the client will accept any certificate sent to it by the server. This includes, specifically, self-signed certificates.

Setting Credentials

There are three primary credentialling methods in HTTPSession:

1. Insert an arbitrary entry into the auth store. Its signature is as follows.

   static public void setAnyCredentialsProvider(HTTPAuthScheme scheme, String url,  CredentialsProvider provider)
   

2. Set it for all sessions, which means it will be applied to any url unless overridden by a more specific entry in the auth store. Its signature is as follows.

   static public void setGlobalCredentialsProvider(HTTPAuthScheme scheme, CredentialsProvider provider)
   

3. Set the credentials on a per-session basis, using the url defined for that session. Its signature is as follows.

   public void setCredentialsProvider(HTTPAuthScheme scheme, CredentialsProvider provider)
   

For backward compatibility, the following two methods are defined. They use the Basic scheme for the scheme.

1. static public void setGlobalCredentialsProvider(CredentialsProvider provider)
   

2. public void setCredentialsProvider(CredentialsProvider provider)
   

Command Line Access to the Authorization Mechanism

Before the introduction of the authorization mechanisms described in this document, it was possible to specify the keystore+password and the truststore+password using the command line flags

-Dkeystore
-Dkeystorepassword
-Dtruststore
-Dtruststorepassword

For purposes of backward compatibility, a check is made for these flags, and if they are defined, then an entry equivalent to the following is inserted into the auth store.

insert(new Entry(HTTPAuthScheme.SSL,ANY_URL,
                 new HTTPSSLProvider(keystore,kpassword,
                                     truststore,trustpassword)));

Miscellaneous Changes

As a side effect of the insertion of the auth mechanisms, a number of miscellaneous changes also occurred.

1. HttpClientManager class was moved to the same package as HTTPSession, namely ucar.nc2.util.net.

Appendices: Session APIs

Note: the following set may lag the actual API, so the JavaDoc or code should be referenced to see any modifications.

HTTPSession API

Static Get/Set Methods

• static synchronized void setGlobalUserAgent(String _userAgent)
  • Set a user agent that will be used when establishing any session.
• static String getGlobalUserAgent()
• static void setGlobalThreadCount(int nthreads)
  • Set the max number of threads that will be supported.
• static int getGlobalThreadCount()
• static Cookie[] getGlobalCookies()
  • Returns the set of cookies available on a generic session.
• static void setGlobalAuthenticationPreemptive(boolean tf)
  • Specifies that all HTTPSession instances should use global preemption
• static void setGlobalSimpleProxy(String host, int port)
  • Force use of a proxy at host:port

Set Credentials Provider(s)

• static void setAnyCredentialsProvider(HTTPAuthScheme scheme, String url, CredentialsProvider provider)
  • Insert an arbitrary entry into the auth store.
  • throws HTTPException
• static void setGlobalCredentialsProvider(HTTPAuthScheme scheme, CredentialsProvider provider)
  • Set provider for all sessions, which means it will be applied to any url unless overridden by a more specific entry in the auth store.
  • throws HTTPException
• public void setCredentialsProvider(HTTPAuthScheme scheme, CredentialsProvider provider)
  • Set the credentials on a per-session basis, using the url defined for that session.
  • throws HTTPException
• static void setGlobalCredentialsProvider(CredentialsProvider cp)
  • For Backward compatibility; this method will create a row in HTTPAuthStore and an instance of HTTPCreds supporting global authorization using the specified credentials provider and the Basic scheme.
  • throws HTTPException
• static void setCredentialsProvider(CredentialsProvider)
  • For Backward compatibility; this method will create a row in HTTPAuthStore and an instance of HTTPCreds supporting authorization for the session's url, using the specified credentials provider and the Basic scheme.
  • throws HTTPException

Constructor(s)

• HTTPSession(String url)
  • Creates a session object with the specified session defining url.
• HTTPSession()
  • Creates a session object compatible with any url.

Note that in both cases, selected command line parameters (specified by the java -D flag) are used for initialization. There are two such cases.

1. Simple Proxy: If defined, the values of the flags "-Dhttp.proxyHost" and "-Dhttp.proxyPort" are used as arguments to the SSL: If defined, the values of the flags "-Dkeystore", "-Dkeystorepassword" and "-Dtruststore", "-Dtruststorepassword" are converted to an HTTPSSLProvider object that in turn is used as an argument to the HTTPMethod API

   Static Methods

   • static HTTPMethod {Get|Head|Put|Post|Options}(HTTPSession session) throws HTTPException
     • Create a method associated with the specified session and session url.
   • static HTTPMethod {Get|Head|Put|Post|Options}(HTTPSession session, String uri) throws HTTPException
     • Create a method associated with the specified session, but using the specified "compatible" url.
   • static void setGlobalMethodParameter(String name, Object value)
     • Set a parameter that is to be applied to all method instance requests.
   • static Enumeration getAllowedMethods()

   Constructor(s)

   • HTTPMethod newMethod{Get|Head|Put|Post|Options}(String uri) throws HTTPException
     • deprecated in favor of the e.g. HTTPMethod.Get factory methods.

   Execute

   • int execute()
     • throws HTTPException

   Responsebody

   The API has a number of methods for getting the response body in various forms.
   • InputStream getResponseBodyAsStream()
   • (aka getResponseAsStream)
   • byte[] getResponseAsBytes(int maxsize)
   • byte[] getResponseAsBytes()
   • String getResponseAsString(String charset) (aka getResponseAsString)

   Instance Get/Set

   • int getStatusCode()
   • String getStatusLine()
   • void setFollowRedirects(boolean tf)

   Request Header Set/Get

   • void setMethodHeaders(List
     headers) throws HTTPException

     void setRequestHeader(String name, String value) throws HTTPException

     void setRequestHeader(Header h) throws HTTPException

     Header getRequestHeader(String name)

     Header[] getRequestHeaders()

   Reponse Header Get

   • Header getResponseHeader(String name)
   • Header[] getResponseHeaders()
   • Header[] getResponseFooters()

   Request Parameter Set/Get

   • void setRequestParameter(String name, Object value)
   • Object getMethodParameter(String key)
   • HttpMethodParams getMethodParameters()

   Request Content Set

   • void setRequestContentAsString(String content) throws HTTPException
   • void setMultipartRequest(Part[] parts) throws HTTPException

   HTTPMethodStream API

   This class subclasses java.io.InputStream and provides the InputStream interface. In addition, if the stream is closed, then the underlying method is closed as well.

   HTTPSSLProvider API

   Since no existing CredentialsProvider implementation exists to support the SSL scheme, the HTTPSSLProvider class is provided for that purpose. Its constructor requires arguments for the client side keystore plus password and the client side truststore plus password. The keystore and truststore arguments are absolute paths.

   Constructor(s)

   • HTTTPSSLProvider(String keystore,String keypass, String truststore, String trustpass)
     • Creates an SSL provider that uses the client side key in the keystore and validates the server-side certificate using the truststore. If the truststore is null, then any certificate presented by the server will be accepted.

   • HTTPSSLProvider()
     • Equivalent to HTTPSSLProvider(null,"",null,"");

   • HTTPSSLProvider(String keystore, String keypass)
     • Equivalent to HTTPSSLProvider(keystore,keypass,null,"");

   Credentials Provider Interface

   • Credentials getCredentials(AuthScheme authscheme, String host, int port, boolean isproxy)
     throws CredentialsNotAvailableException
     • This function is here only to satisfy the interface and will never be invoked.

   Author

   Author: Dennis Heimbigner
   Affiliation: UCAR/Unidata
   email: dmh@ucar.edu