Unidata

System requirements

• Sun Java 1.6 (latest version)
• Apache Tomcat 5.5 or above (we recommend the latest version of Tomcat 6.0)

While there are different distributors of Java and servlet containers, Unidata develops, uses and tests the THREDDS Data Server using Sun Java and the Apache Tomcat servlet container.

Installing Java

1. Download current Java SE Developer Kit (JDK) from Sun.

At the time of this workshop, that is version 1.6.0_26. We will be using the Linux x64-bit, non-rpm, self-extracting file:

jdk-6u26-linux-x64.bin

2. Install the JDK as per the Sun installation instructions.

Copy the self-extracting file to the installation directory (~/GettingStarted is the installation directory in this example):

$ pwd
/home/tds

$ cp Downloads/jdk-6u26-linux-x64.bin GettingStarted/ 
$ cd GettingStarted

Change the permissions on the file to be executable:

$ ls -l  
-rw-r--r-- 1 tds Unidata 85407904 Jul 11 17:26 jdk-6u26-linux-x64.bin

$ chmod 775 jdk-6u26-linux-x64.bin  

$ ls -l  
-rwxrwxr-x 1 tds Unidata 85407904 Jul 11 17:26 jdk-6u26-linux-x64.bin

Run self-extracting file and agree to the terms of the code license:

$ ./jdk-6u26-linux-x64.bin 

This will extract the JDK in the ~/GettingStarted directory:

$ ls -l 
drwxr-xr-x 10 tds Unidata     4096 Jul 11 17:35 jdk1.6.0_26
-rwxrwxr-x  1 tds Unidata 85407904 Jul 11 17:26 jdk-6u26-linux-x64.bin

Installing Tomcat

1. Download current version of the Tomcat servlet container.

At the time of this workshop, that is version 6.0.32. We will be using the core binary tar.gz file:

apache-tomcat-6.0.32.tar.gz

2. Install Tomcat as per the Apache Tomcat installation instructions.

Copy the binary tar.gz file to the installation directory (~/GettingStarted is the installation directory in this example):

$ pwd
/home/tds/GettingStarted

$ cp ~/Downloads/apache-tomcat-6.0.32.tar.gz . 

Unpack the archive to create Tomcat directory:

$ tar xvzf apache-tomcat-6.0.32.tar.gz

$ ls -ld apache-tomcat-6.0.32
drwxr-xr-x 9 tds Unidata 4096 Jul 11 17:47 apache-tomcat-6.0.32

Exploring the Tomcat directory structure

1. Examine the Tomcat directory structure, aka $CATALINA_HOME (also can be called $TOMCAT_HOME).

Move into $CATALINA_HOME and do a long listing:

$ cd apache-tomcat-6.0.32
$ ls -l 
total 92
drwxr-xr-x 2 tds Unidata  4096 Jul 11 17:47 bin
drwxr-xr-x 2 tds Unidata  4096 Feb  2 12:06 conf
drwxr-xr-x 2 tds Unidata  4096 Jul 11 17:47 lib
-rw-r--r-- 1 tds Unidata 37951 Feb  2 12:06 LICENSE
drwxr-xr-x 2 tds Unidata  4096 Feb  2 12:04 logs
-rw-r--r-- 1 tds Unidata   558 Feb  2 12:06 NOTICE
-rw-r--r-- 1 tds Unidata  8672 Feb  2 12:04 RELEASE-NOTES
-rw-r--r-- 1 tds Unidata  6670 Feb  2 12:06 RUNNING.txt
drwxr-xr-x 2 tds Unidata  4096 Jul 11 17:47 temp
drwxr-xr-x 7 tds Unidata  4096 Feb  2 12:04 webapps
drwxr-xr-x 2 tds Unidata  4096 Feb  2 12:04 work

2. Familiarize yourself with the following important directories.

bin/

• Contains startup.sh, shutdown.sh and other scripts/programs.
• The *.sh files (for Unix systems) are functional duplicates of the *.bat files (for Windows systems).

conf/

• Server-wide Tomcat configuration.
• You will modify server.xml and tomcat-users.xml to adjust logging, authentication and access control, enable SSL, etc.
• Web applications can override some server-wide settings in their own configuration files (more about that later).

webapps/

• Contains web applications directories and WAR files.
• This is where we will be putting the TDS web application.
• You will also be using the manager application that comes with Tomcat during this workshop.

logs/

• Tomcat log files are here by default.
• This is one of the directories you will be looking for log files (the TDS logs elsewhere by default).
• The log files should be your first stop for troubleshooting Tomcat and TDS issues. (Hint, hint.)
• Logs files may contain useful information for assessing the security of your system.
• You will become very familiar with the Tomcat-generated catalina.out and localhost_access_log.yyyy-mm-dd.log files by the end of this workshop.

Starting & stopping Tomcat

1. Tomcat isn't currently running so we need to start it up.

Run the startup.sh script in the Tomcat bin/ directory:

$ pwd
/home/tds/GettingStarted/apache-tomcat-6.0.32

$ bin/startup.sh

2. Verify Tomcat is running.

Look and see if you have a Tomcat process running:

$ ps -ef | grep tomcat
tds      27049     1  2 17:50 pts/1    00:00:02 /usr/bin/java -Djava.util.logging.config.file=/home/tds/GettingStarted/apache-tomcat-6.0.32/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/home/tds/GettingStarted/apache-tomcat-6.0.32/endorsed -classpath /home/tds/GettingStarted/apache-tomcat-6.0.32/bin/bootstrap.jar -Dcatalina.base=/home/tds/GettingStarted/apache-tomcat-6.0.32 -Dcatalina.home=/home/tds/GettingStarted/apache-tomcat-6.0.32 -Djava.io.tmpdir=/home/tds/GettingStarted/apache-tomcat-6.0.32/temp org.apache.catalina.startup.Bootstrap start

Open a new browser window/tab and go to http://localhost:8080/ to verify Tomcat is running:



Which Java is Tomcat currently using? (Hint: what was sent to STDOUT when running the startup.sh and shutdown.sh?)

3. See if you can shutdown Tomcat.

Run the shutdown.sh script in the Tomcat bin/ directory:

$ bin/shutdown.sh

Setting $JAVA_HOME, $JAVA_OPTS and $CATALINA_HOME

We are going to create a file called setenv.sh in the Tomcat bin/ directory to:

• allow Tomcat to reference/find the location of $JAVA_HOME and $CATALINA_HOME) during startup and shutdown; and
• increase the amount of memory allocated to the JVM to enhance performance by setting $JAVA_OPTS.

1. Create the setenv.sh file.

Using your favorite text editor (gedit, vi, emacs, etc.), create a new file called setenv.sh in the Tomcat bin/ directory:

$ pwd
/home/tds/GettingStarted/apache-tomcat-6.0.32

$ cd bin
$ vi setenv.sh

Add the following information and save your setenv.sh file:

#!/bin/sh
#
# ENVARS for Tomcat and TDS environment
#
JAVA_HOME="/home/tds/GettingStarted/jdk1.6.0_26"
export JAVA_HOME

JAVA_OPTS="-Xmx4096m -Xms512m -server -Djava.awt.headless=true -Djava.util.prefs.systemRoot=$CATALINA_HOME/content/thredds/javaUtilPrefs"
export JAVA_OPTS

CATALINA_HOME="/home/tds/GettingStarted/apache-tomcat-6.0.32"
export CATALINA_HOME

Settings

Whenever possible, Unidata recommends -Xmx1500m for 32-bit systems, and-Xmx4096m (or more) for 64-bit systems.

The parameters we pass to $JAVA_OPTS:

• -Xms is the initial allocated memory of the JVM (for performance).
• -Xmx the maximum allocated memory of the JVM (for performance).
• -server tells the Hostspot compiler to run the JVM in "server" mode.
• -Djava.awt.headless=true is needed to prevent graphics rendering code from assuming a graphics console exists. Without this, WMS code will crash the server in some circumstances.
• -Djava.util.prefs.systemRoot=$CATALINA_HOME/content/thredds/javaUtilPrefs allows the java.util.prefs of the TDS WMS service to write system preferences to a location that is writable by the user that Tomcat runs under
.
2. Restart Tomcat to implement your changes.

Restart Tomcat and examine the output generated to the terminal window by the startup script:

$ ./startup.sh
Using CATALINA_BASE:   /home/tds/GettingStarted/apache-tomcat-6.0.32
Using CATALINA_HOME:   /home/tds/GettingStarted/apache-tomcat-6.0.32
Using CATALINA_TMPDIR: /home/tds/GettingStarted/apache-tomcat-6.0.32/temp
Using JRE_HOME:        /home/tds/GettingStarted/jdk1.6.0_26
Using CLASSPATH:       /home/tds/GettingStarted/apache-tomcat-6.0.32/bin/bootstrap.jar

Did you notice any difference in the what is being reported to STDOUT during startup?

Take a look at the running Tomcat process to see the new $JAVA_OPTS settings:

$ ps -ef | grep tomcat
tds       2697     1  7 17:59 pts/1    00:00:03 /home/tds/GettingStarted/jdk1.6.0_26/bin/java -Djava.util.logging.config.file=/home/tds/GettingStarted/apache-tomcat-6.0.32/conf/logging.properties -Xmx1500m -Xms512m -server -Djava.awt.headless=true -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/home/tds/GettingStarted/apache-tomcat-6.0.32/endorsed -classpath /home/tds/GettingStarted/apache-tomcat-6.0.32/bin/bootstrap.jar -Dcatalina.base=/home/tds/GettingStarted/apache-tomcat-6.0.32 -Dcatalina.home=/home/tds/GettingStarted/apache-tomcat-6.0.32 -Djava.io.tmpdir=/home/tds/GettingStarted/apache-tomcat-6.0.32/temp org.apache.catalina.startup.Bootstrap start

What allows us to create the setenv.sh file and have its contents read? (Hint: have a look at the catalina.sh file in the Tomcat bin/ directory starting around line 117).

Troubleshooting tips

• Do not forget include the 'm' in your -Xms and -Xmx settings.
• You may have allocated too much memory for the JVM settings if Tomcat fails to start and you get the following error reported in the Tomcat log catalina.out:

Error occurred during initialization of VM
Could not reserve enough space for object heap

• Likewise, if there is an error with your JVM memory allocation syntax in the setenv.sh file, it will be reported to catalina.out:

Error occurred during initialization of VM
Incompatible minimum and maximum heap sizes specified

About WAR files

• WAR is short for Web ARchive.
• By default, Tomcat will automatically unpack the WAR distribution into directory of the same name upon deployment.
• Note: the unpacked directory is overwritten each time a new WAR file is deployed.

Downloading & deploying thredds.war

1. Download the TDS WAR file from Unidata's web site.

We will be using the current TDS 4.2 (stable) version for this workshop:

thredds.war

2. Deploy the TDS in Tomcat.

Put thredds.war in the Tomcat webapps/ directory:

$ pwd
/home/tds/GettingStarted/apache-tomcat-6.0.32/bin
$ cd ../webapps

$ cp ~/Downloads/thredds.war .
$ ls -l
total 24612
drwxr-xr-x 10 tds Unidata     4096 Jul 11 17:47 docs
drwxr-xr-x  5 tds Unidata     4096 Jul 11 17:47 examples
drwxr-xr-x  5 tds Unidata     4096 Jul 11 17:47 host-manager
drwxr-xr-x  5 tds Unidata     4096 Jul 11 17:47 manager
drwxr-xr-x  3 tds Unidata     4096 Jul 11 17:47 ROOT
-rw-r--r--  1 tds Unidata 25238961 Jul 11 18:02 thredds.war

3. Confirm the TDS has been deployed.

If Tomcat is already running, wait a couple of seconds after placing the WAR file in the Tomcat webapps/ and then verify the thredds.war file was unpacked:

$ ls -l
total 24616
drwxr-xr-x 10 tds Unidata     4096 Jul 11 17:47 docs
drwxr-xr-x  5 tds Unidata     4096 Jul 11 17:47 examples
drwxr-xr-x  5 tds Unidata     4096 Jul 11 17:47 host-manager
drwxr-xr-x  5 tds Unidata     4096 Jul 11 17:47 manager
drwxr-xr-x  3 tds Unidata     4096 Jul 11 17:47 ROOT
drwxr-xr-x  6 tds Unidata     4096 Jul 11 18:02 thredds
-rw-r--r--  1 tds Unidata 25238961 Jul 11 18:02 thredds.war

Go to http://localhost:8080/thredds/ in your browser to verify the TDS has been deployed:

Tomcat logs/

1. Look at the different types of log files being generated in the Tomcat logs/ directory.

Move into the logs/ directory to see the type of information being logged:

$ pwd
/home/tds/GettingStarted/apache-tomcat-6.0.32/webapps
$ cd ../logs

$ ls -l
total 20
-rw-r--r-- 1 tds Unidata 4423 Jul 11 18:02 catalina.2011-07-11.log
-rw-r--r-- 1 tds Unidata 4741 Jul 11 18:02 catalina.out
-rw-r--r-- 1 tds Unidata    0 Jul 11 17:50 host-manager.2011-07-11.log
-rw-r--r-- 1 tds Unidata 2132 Jul 11 18:02 localhost.2011-07-11.log
-rw-r--r-- 1 tds Unidata    0 Jul 11 17:50 manager.2011-07-11.log

Do you see a correspondence between some of the web applications in the Tomcat webapps/ directory and the naming of certain log files?

Is there a difference in the information being logged to catalina.out versus catalina.yyyy-mm-dd.log?

Are some log files more verbose than others?

2. Examining catalina.out.

Open another terminal window (hereafter referred to as terminal #2) and run the following command in the new terminal:

$ tail -f /home/tds/GettingStarted/apache-tomcat-6.0.32/logs/catalina.out

In your original terminal window, start/stop and start Tomcat and watch what is being logged to catalina.out in the terminal #2 window.

Is it only errors messages being reported to catalina.out?

What messages in catalina.out are from the TDS?

Things to know about catalina.out

• Tomcat System.out and System.err gets appended to catalina.out.
• catalina.out can quickly grow large if the hosted web applications are not specifically catching and logging System.out and System.err to designated files.
• catalina.out is not automatically rotated in Tomcat.
• You should employ an outside log rotation program (e.g., logadm or logrotate) to rotate catalina.out.
• It is good practice to archive and remove old catalina.out files and other log files out of the Tomcat logs/ on a regular basis.

About server.xml

• XML file (well-formed syntax is important).
• Tomcat's main configuration file.
• Changes to server.xml do not take effect until Tomcat is restarted.
• Where we make changes to enhance TDS security.

Important elements in server.xml

1. Examine the Elements in server.xml.

Move into the Tomcat conf/ directory and examine the server.xml file:

$ pwd
/home/tds/GettingStarted/apache-tomcat-6.0.32/logs
$ cd ../conf

$ less server.xml

Reference the table below to see how the server.xml elements relate to configuring TDS (mouse-over the element for a description):

Tag Name	Instances	How it relates to the TDS
<Server>Description: The Server element represents the entire Catalina servlet container as a whole. It is the single outermost element in server.xml	1...1	Not modified unless you want to change the port number Tomcat listens for a SHUTDOWN command. (Enabled by default.)
<GlobalNamingResources>Description: The GlobalNamingResources element defines the global Java Naming and Directory Interface (JNDI) resources for the Server.	0...*	Needed to contain the UserDatabase that corresponds to the UserDatabaseRealm used to authenticate users. (Enabled by default.)
<Resource>Description: The Resource element represents a static resource from which classes will be loaded and static files will be served.	0...*	Editable user database (tomcat-users.xml) used by UserDatabaseRealm to authenticate users. (UserDatabaseRealm Resource enabled by default.)
<Service>Description: The Service element represents the combination of one or more Connector components that share a single Engine component for processing incoming requests. The top Tomcat service is named Catalina (hence the log file name of catalina.out).	1...*	Not modified unless you wish to establish more than one service. (Catalina Service enabled by default.)
<Connector>Description: The Connector element forward requests to the Engine using a specific protocol and returns the results to the requesting client.	1...*	Used to establish HTTP and SSL connections. Also will communicate with an web server for proxying requests. (HTTP connector enabled by default on port 8080.)
<Engine>Description: The Engine element represents the entire request processing machinery associated with a particular Catalina Service.	1...1	Not modified unless you specify a Host other than localhost. (Enabled by default.)
<Realm>Description: The Realm element represents a "database" of usernames, passwords, and roles (groups) assigned to those users.	0...*	The UserDatabaseRealm uses the UserDatabase configured in the global JNDI Resource. (UserDatabaseRealm enabled by default.)
<Valve>Description: The Valve element represents a component that will be inserted into the request processing pipeline for the associated containing element.	0...*	The RemoteAddrValve is used to filter access to the TDS based on IP address. (NOT enabled by default. You will need to add this if you want to use IP Filtering.)
<Host>Description: The Host element represents a virtual host.	1...*	Not modified unless you specify a Host other than localhost. (localhost enabled by default.)
<Realm>Description: The Realm element represents a "database" of usernames, passwords, and roles (groups) assigned to those users.	0...*	We use the MemoryRealm to configuring Tomcat to use digested passwords. (NOT enabled by default. You will need to add this if you want to use digested passwords.)
<Valve>Description: The Valve element represents a component that will be inserted into the request processing pipeline for the associated containing element.	0...*	We modify the AccessLogValve to customize the access logs generated by Tomcat. (NOT enabled by default. You will need to add this if you want to enable access logging.)

About tomcat-users.xml

• XML file (well-formed syntax is important).
• Stores user names, passwords and roles.
• Changes to tomcat-users.xml do not take effect until Tomcat is restarted.
• What the TDS uses for user authentication and access control.

Important elements in tomcat-users.xml

1. Examine the Elements in tomcat-users.xml.

Open the tomcat-users.xml file:

$ pwd
/home/tds/GettingStarted/apache-tomcat-6.0.32/conf

$ less tomcat-users.xml

Reference the table below to see how the tomcat-users.xml elements relate to configuring TDS (mouse-over the element for a description):

Tag Name	Instances	How it relates to the TDS
<tomcat-users>Description: The tomcat-users element represents the single outermost element in tomcat-users.xml	1...1	Not modified. (The only tag you get by default.)
<role>Description: The role element defines one role or group a user can belong to.	1...*	You will have at least two of these: one for the Tomcat manager application and one for the TDS. (You will need to add if you want to enable role-based authentication.)
<user>Description: The user element represents one valid user.	1...*	You will need to create an entry for each user who needs access to the Tomcat manager application and/or the restricted areas of the TDS. (You will need to add if you want to enable user authentication.)

About the manager application

• "Free" web application that comes with Tomcat distribution.
• Lives in the manager directory in the Tomcat webapps/ directory.
• Allows Tomcat administrators to deploy, undeploy, or reload web applications such as the TDS without having to shut down and restart Tomcat.
• Provides server status statistics for the JVM and each connector you have configured in server.xml.

Accessing the Tomcat manager application

Attempt to access the Tomcat manager application in your browser: http://localhost:8080/manager/html/. You will be prompted to login via BASIC authentication, which will end in failure since we do not yet have permission to access the manager application:

Granting access to the manager application

1. Modify tomcat-users.xml to add role and user elements.

Using your favorite editor, open $TOMCAT_HOME/conf/tomcat-users.xml:

$ vi tomcat-users.xml

Between the <tomcat-users> tags, add a role element and specify the rolename attribute as manager:

<tomcat-users>
    <role rolename="manager-gui"/>
</tomcat-users>

Now add a new user by adding a user element. Create a username and password for the new user and specify manager-gui as one of the roles (in this example we are creating a user called 'admin' with a corresponding password of 'secret'):

<tomcat-users>
    <role rolename="manager-gui"/>
    <user username="admin" password="secret" roles="manager-gui"/>   
</tomcat-users>

2. Restart Tomcat and log into the manager application.

Keep in mind

Changes to tomcat-users.xml do not take effect until Tomcat is restarted.

Attempt to access the manager application again (http://localhost:8080/manager/html/), this time logging in using the name and password specified in tomcat-users.xml:



Thinking ahead

To gain access to restricted parts of the TDS, you will perform the same steps you used to grant yourself access to the manager application.

Voilà! You should have access to the manager application.

Deploying the TDS using the manager application

1. Use the manager application to undeploy the TDS.

Find the TDS in the list of web application on the Applications page. Stop and then Undeploy the TDS:



List the contents of the Tomcat webapps/ directory to verify that both thredds.war and the unpacked thredds/ directory have been removed:

$ pwd
/home/tds/GettingStarted/apache-tomcat-6.0.32/conf
$ cd ../webapps

$ ls -l
total 20
drwxr-xr-x 10 tds Unidata 4096 Jul 11 17:47 docs
drwxr-xr-x  5 tds Unidata 4096 Jul 11 17:47 examples
drwxr-xr-x  5 tds Unidata 4096 Jul 11 17:47 host-manager
drwxr-xr-x  5 tds Unidata 4096 Jul 11 17:47 manager
drwxr-xr-x  3 tds Unidata 4096 Jul 11 17:47 ROOT

2. Deploy the TDS using the manager application.

Upload the TDS WAR file using the Deploy section of the manager application:



Confirm the deployment went as planned by accessing the TDS using your browser: http://localhost:8080/thredds/

Caveat of the manager application

The dreaded java.lang.OutOfMemoryError: PermGen space failure error:

• The issue: The "PermGen" error happens when the JVM runs out of memory in the permanent generation. (Default PermGen size differs among Sun JDK versions, but the upper limit is typically 64MB.)
• The cause: Objects in the permanent generation are never garbage collected. When redeploying your web application using the Tomcat manager application, your WAR file is unpacked and parts of the class file definition are loaded into PermGen space, like string constants.
• The symptom: The PermGen error will manifest itself in a sluggish Tomcat manager application that never completes a task, and the java.lang.OutOfMemoryError: PermGen space failure error being displayed in $TOMCAT_HOME/logs/catalina.out
• A temporary fix: You can add the -XX:MaxPermSize switch to $JAVA_OPTS to increase the amount of memory allocated for the permanent generation, However this is only postponed the inevitable, as even an increased memory in permanent generation will eventually fill up. When this happens, you will need to stop/start Tomcat at this point. For this reason, you may want to restart Tomcat whenever you redeploy TDS or another webapp.

You're not finished yet

• We highly recommend you follow the best practices outlined in the Tomcat and TDS Security Considerations section of the tutorial to finish hardening your server environment.
• The Basic TDS Configuration and TDS Configuration Catalogs sections of this tutorial covers the TDS configuration files, configuration options and TDS catalog structure.
• Additional sections of the tutorial cover how to enable OGC/ISO Services, and Using NcML in TDS.