UnidataTHREDDS Data Server (TDS) FAQ


General

Catalogs

TDS

Caching

TDS Install Errors

Tomcat


General

Q: I have a strange problem and I need help figuring out whats going on!

A: Here is what we need from you in order to deal with hard problems:

  1. Run the latest stable release. Sorry, we dont have the resources to keep older versions running, so we need to deal with just the latest code.
  2. Get a clean set of logs that capture the problem:
    1. Stop the tomcat server.
    2. Install the latest release if needed.
    3. Remove all files from {tomcat}/logs and {tomcat}/content/thredds/logs
    4. Restart the server
    5. Make the problem happen
    6. Zip up everything in {tomcat}/logs and {tomcat}/content/thredds/logs.
    7. Send the logs and a detailed description of what you do to make the problem happen, and what the problem looks like. If it took a while to get the problem to happen, note what time it happened so that we can correlate with the logs.

Q: I made changes to my catalog and restarted tomcat but nothing changes.

  1. look in catalina.out for a message that tomcat did not shut down:
    java.net.BindException: Address already in use:8080 
  2. make sure tomcat really gets stopped:
    1. ps -ef | grep java to find the process id
    2. kill <pid> or kill -9 <pid>
    3. ps -ef | grep java to verify that the process goes away.
  3. Restart tomcat:
    1. sh ./startup.sh
    2. Check catalina.out that tomcat started correctly
    3. ps -ef | grep java to verify that theer is a new tomcat process.

Q: I'm building my own maven project and want to use your jar files. Where is the Unidata Repository?

A: In order to configure your maven project correctly, you need to edit your pom.xml file to reflect the location of Unidata's repository. To do this, you need to add the following pom snippet to your pom.xml file:

<distributionManagement>
    <repository>
        <id>unidata-releases</id>
        <name>UNIDATA Releases</name>
        <url>https://artifacts.unidata.ucar.edu/content/repositories/unidata-releases/</url>
    </repository>
</distributionManagement>

Alternatively, you can configure your settings.xml file to reflect the repository location. Information on doing either is located here:

  Maven Repository information

and here:

  Maven Settings information


Catalogs

Q: How do I construct the URLs I find in a THREDDS Catalog?

A: Heres the general idea in the tutorial and the reference docs. If you are using the CDM library, you can call

InvAccess.getUrlPath()

Q: How do I eliminate the extra dataset when using a Catalog Reference?

Make the name of the catalogRef the same as the "top" dataset in the referenced catalog. In the following example the name is ""New Point Data".

In the referencing catalog:

 <catalogRef xlink:href="idd/newPointObs.xml" xlink:title="New Point Data" name="" />

In the referenced catalog:

<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0" 
 xmlns:xlink="http://www.w3.org/1999/xlink" name="THREDDS-IDD OPeNDAP Data Server - New Station Data"
 version="1.0.3">

  <service name="ncdods" serviceType="OPENDAP" base="/thredds/dodsC/"/>
  <dataset name="New Point Data">
    ...
  </dataset>
</catalog>

TDS

Q: How do I keep client requests from overwhelming my server?

There is currently no per-client resource throttle, unfortunately, but we are aware of the need for that. Any given request is single-threaded, so can't hog too many resources. One can limit the size of opendap responses, which tends to be the main problem on some servers. See the OPeNDAP section of the threddsConfig.xml page for details.


Q: Can aggregations of many files cause "too many files open" problems?

Aggregations only open one file at a time, and then close it, so this can't cause "too many file" problems.

If you have "too many open files" errors, then either theres a file leak (which we would like to know about), or you have your file cache limit set too high relative to your OS file handle limit.


Q: What do the non-HTTP status codes in the threddsServlet.log files mean?

The "Request Completed" messages in the threddsServlet.log files contain several fields including a status code. the HTTP status code returned in a completed response. If a request is forwarded to another internal service, a "1000 (Forwarded)" or "1001 (

2009-06-17T13:25:54.451 -0600 [     28949][      11] INFO
- thredds.server.catalogservice.LocalCatalogServiceController
- handlePublicDocumentRequest(): Request Completed - 1001 - -1 - 32

Q: I'm seeing the error "Inconsistent array length read: 538976288 != 1668244581' when I open the dataset in the IDV. Why?

The error "Inconsistent array length read" only tells you that there was an error on the server in the middle of responding to an OPeNDAP request. You then must look in the threddsServlet.log and find the error to know why.


Q: Why am I getting lots of java.util.prefs.BackingStoreException warning messages?

If you allow and use the TDS WMS service, you may be seeing warning messages in your Tomcat catalina.out log file that look something like this:

May 25, 2010 6:28:22 PM java.util.prefs.FileSystemPreferences syncWorld
WARNING: Couldn't flush system prefs: java.util.prefs.BackingStoreException: /etc/.java/.systemPrefs/org create failed.

You can get rid of these messages by setting the "java.util.prefs.systemRoot" system property to a location that is writable by the user that Tomcat runs under.

Here is what we do on our servers:

  • Create a directory at ${tomcat_home}/content/thredds/javaUtilPrefs/.systemPrefs, e.g.,
    cd ${tomcat_home}/content/thredds
    mkdir javaUtilPrefs
    mkdir javaUtilPrefs/.systemPrefs
    
    mkdir javaUtilPrefs/.usermPrefs
    
  • Make sure that the.systemPrefs and .userPrefs directories are writable by the user under which Tomcat runs
  • Add the following to JAVA_OPTS in the ${tomcat_home}/bin/setenv.sh file:
    -Djava.util.prefs.systemRoot=$CATALINA_HOME/content/thredds/javaUtilPrefs -Djava.util.prefs.userRoot=$CATALINA_HOME/content/thredds/javaUtilPrefs

If you are interested in more details of the problem, here are two useful links:

We have this TDS issue in our bug tracking system and plan to address it.


Q: My TDS server is behind a proxy server. Why do some TDS generated URLs point to my TDS server instead of my proxy server?

Most TDS generated URLs are relative to the server (e.g., "/thredds/dodsC/") or relative to the the current document's base URL. There are only a few places where it is necessary to generate absolute URLs. In those cases, the TDS uses information from the incoming HTTP request to construct the generated URLs. It is up to the proxy to send the correct information to the proxied server so the request information will be correct.

For more information, see our web page on running Tomcat behind a proxy server. It contains links to Tomcat documentation on both mod_proxy and mod_jk as well as some user contributed documentation on setting up mod_proxy.


Q: I have modified my configuration of a JoinExisting Aggregation dataset, but nothing has changed.

The files and coordinates in a JoinExisting Aggregations are cached, and in some circumstances won't get updated. The default location for the cache is ${tomcat_home}/content/thredds/cache/agg/ unless you change it in the threddsConfig.xml file. Go to that directory, there will be files with the name of the cached dataset(s). Delete the file for the dataset that needs updating and restart Tomcat.


Q: What happened to the long list of CRSs in my WMS GetCapabilities documents?

In TDS 4.1, each WMS GetCapabilities document listed 100s of available CRS. This made for very large GetCapabilities documents. As of TDS 4.2, this list is limited to a select few CRSs. We hope to make this list configurable in a future release.

In the mean time if you need a CRS that isn't listed, try specifying it in the GetMap request. The underlying library that handles CRS (Geotoolkit) still supports a large number of CRS and the TDS WMS should still support any of those CRS when requested.


Q: Why are TDS web forms not working?

Look in {$TOMCAT}/logs/localhost.logs for error messages like:

     SEVERE: Servlet.service() for servlet GridSubsetService threw exception
   javax.xml.transform.TransformerFactoryConfigurationError: Provider net.sf.saxon.TransformerFactoryImpl not found

If you find these, the likely problem is that another webapp running in the same Tomcat container has set the XSLT parser with javax.xml.transform.TransformerFactory, which is global for the JVM. The above example shows that the Saxon parser has been set, but is not being found by the TDS. We saw this happening with the OOSTethys webapp.

The solution is to move the other webapp to its own Tomcat instance, or to move the required jar (eg saxon.jar) into Tomcat's lib directory, where it is available to all webapps. TDS does very simple XSLT to create its web forms, so its likely that it can work with any decent XSLT library. By default it uses the JDK's built-in XSLT library.


Q: What does the TDS do at startup to read the configuration catalogs? What gets cached? Does it have a way to know a referenced catalog is unchanged? When do referenced catalogs get scanned?

The TDS reads in all the config catalogs at startup. It caches all of them, and uses the "expires" attribute on the catalog to decide if/when it needs to re-read a catalog.  It must read all catalogs, including catalogRefs, because it has to know what the possible dataset URLs are, and there is no contract that a client has to read a catalog before accessing the dataset.


Q: Why do I get a message that "class org.apache.catalina.ant.DeployTask cannot be found" when I try to build the TDS from source using Ant?

Note: Ant builds are not supported in TDS 4.3, and build.xml is removed from the repository as of version 4.3.16

The TDS build.xml file uses several external Ant tasks defined in the catalina-ant.jar file. For TDS 4.2.5 and before, you need to have the catalina-ant.jar file in the lib/ directory of your Ant installation. If it is not there you will get a message that looks something like:

../tds/build.xml:730: taskdef class org.apache.catalina.ant.DeployTask cannot be found

You can find the catalina-ant.jar file in the lib/ directory of a Tomcat distribution.

For TDS 4.2.6 and above, you should not see this problem as the build.xml file directly references a copy of the catalina-ant.jar file.

For TDS 4.3 and above, building with ant is no longer supported. Build with maven instead.


Caching

Q: We use compressed netcdf files and the very first access to them are quite slow, although subsequent accesses are much faster, then become slow again after a while. I can see that TDS uncompress these files to the cdm cache directory, but then they must get deleted. Is there a way to keep them in the cache permanently?

Esentially this is a tradeoff between storage spaceand the time to decompress. I assume you dont want to store the files uncompressed, so you have to pay the price of that. To control how these files are cached, see CDM library Disk cache. I would suggest that you use:

 <DiskCache>
<alwaysUse>true</alwaysUse>
<scour>1 hour</scour>
<maxSize>10 Gb</maxSize>
</DiskCache>

and choose maxSize carefully. The trick is to make maxSize big enough to keep the "working set" uncompressed, i.e. if there is a reletively small "hot" set of files that get accessed a lot, you want to give enough cache space to keep them uncompressed in the cache. Monitor the cache directory closely to see what files stay uncompressed, and how old they are, and modify maxSize as needed.


TDS Install Errors

Q: Why does the TDS fail to deploy in the Tomcat provided by RedHat 5 or 6?

The Tomcat installation provided with RedHat 5 and 6 (and with yum) is run from /usr/share/tomcat5 which contains symbolic links that point into /var/lib/tomcat5. The permissions are such that the TDS cannot write the content/thredds directory in the standard location, ${tomcat_home}/content/thredds (in this case /usr/share/tomcat5/content/thredds). The error message in catalina.out looks something like this:

Jun 17, 2009 3:44:08 PM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deploying web application archive thredds.war
log4j:WARN No appenders could be found for logger (org.apache.commons.digester.Digester.sax).
log4j:WARN Please initialize the log4j system properly.
TdsConfigContextListener.contextInitialized(): start.
Jun 17, 2009 3:44:11 PM org.apache.catalina.core.StandardContext start
SEVERE: Error listenerStart
Jun 17, 2009 3:44:11 PM org.apache.catalina.core.StandardContext start
SEVERE: Context [/thredds] startup failed due to previous errors
  

And there should be a message in the localhost.*.log file that looks something like this:

ERROR - TdsContext.init(): Content directory does not exist and could not be created [/usr/share/tomcat5/content]

There are currently two solutions:

  1. Create the needed directories by hand and setup appropriate symlinks
    • create the TDS content directory in /var/lib/tomcat5;
    • make sure the tomcat user owns the content directory and has read/write permission; and
    • create the symlink /usr/share/tomcat5/content that points to the directory created above.
  2. Give the TDS an absolute path to the desired location for the content directory by setting the tds.content.root.path system property with a command-line argument (e.g., "-Dtds.content.root.path=/some/absolute/path"). More information is available here.

Q: ERROR - TdsContext.init(): Content directory does not exist and could not be created

The TDS needs to create the directory ${tomcat_home}/content but it does not have permission, typically because ${tomcat_home} is owned by root. Make ${tomcat_home} owned by the tomcat user, or manually create ${tomcat_home}/content and make it owned by the tomcat user.


Q: On starting up TDS, I get the error "SEVERE: Error listenerStart" and "SEVERE: Context [/thredds] startup failed due to previous errors", and TDS wont start.

where:

  1. Harmless log4j warnings. Someday we'll figure out how to get rid of it.
  2. Various initialization info messages
  3. This is the problem, but it doesnt actually contain enough information to know whats going on. It usually means theres an error in how you set up Tomcat.

Q: What does this error mean: "log4j:ERROR Attempted to append to closed appender named [foobar]" ?

The log4j.xml file has 2 loggers with the same name, that uses the appender "foobar". You must delete one of the loggers.


Tomcat

Q: Im getting the error "java.lang.OutOfMemoryError: PermGen space". Whats up?

If you reload the thredds.war webapp enough times without restarting Tomcat, you will eventually run into "java.lang.OutOfMemoryError: PermGen space". This is a known bug in JDK/Tomcat. The only thing to do is to stop and restart Tomcat.

You can increase PermGen using this JVM option:

-XX:MaxPermSize=180m

On the JDK1.6 -server JVM, the default seems to be 64m. However, with enough redeploys , you will eventually run out of PermGen space no matter what your MaxPermSize setting is. We have gotten into the habit of restarting tomcat on our production server whenever we redeploy. Lots of redeploys only happen on our test server.

Resources:


Q: During shutdown I'm getting messages about threads (ThreadLocal) having to be shut down to prevent memory leaks. Whats up?

Tomcat memory leak detection code started logging these messages as of Tomcat 6.0.24. From various posts (see Spring Forum: "ThreadLocal forcefully removed" comment #3 e.g.) it appears that these messages are not a problem but instead a matter of Tomcat finding these objects before they get garbage collected.

Here are a number of related links:

NOTE: We will monitor the status of this Tomcat issue. For now, we do not consider this a TDS bug and will not be working to fix this issue in TDS.


Q: Who is accessing my server?

When you examine the TDS access logs, you can see who is accessing the TDS by IP address. Use nslookup <ip address> to find out the host name.


Q: How can I control whether I want Web crawlers to access my server?

Well-behaved web crawlers are supposed to look for a robots.txt file on the server and follow its instructions. To set up a robots.txt file that excludes web crawlers from crawling your server, follow these directions.


Q: How can I prevent someone from accessing my server?

If your server is being overwhelmed by requests from a particular user/computer, it is best to exclude them using their IP address rather than their hostname (this avoids having to perform a DNS lookup for each request). To do so, edit the ${tomcat_home}/conf/server.xml file and find the "localhost" Host element. Add a RemoteAddrValve Valve element as follows:

<Host name="localhost" debug="0" appBase="webapps" .. >
  <Valve className="org.apache.catalina.valves.RemoteAddrValve" deny="18\.83\.0\.150" />
  ...
</Host>
  

The value of the deny attribute must be one or more (comma delimited) regular expressions each of which will be compared to the remote clients IP addresses. For instance:

deny="18\.83\.0\.150,128\.100\.34\.99,128\.117\.140\..*"
  

NOTE: You need to restart the server before this will take effect.


Q: How do I remove Servlet Autodeploy?

Its recommended to remove autodetection of changes while Tomcat is running, for performance reasons. In a production environment, its better to explicitly redeploy the application:

   <Host name="localhost" appBase="webapps" unpackWARs="true" autoDeploy="false"
     xmlValidation="false" xmlNamespaceAware="false">
   ...
  </Host>

Q: How do I remove port 8009 when using tomcat in standalone mode?

Unless you are using Tomcat with the Apache server, comment out this line in server.xml:

  <Connector port="8009" enableLookups="false" redirectPort="8443" protocol="AJP/1.3" />

Tomcat Resources


THREDDS This document is maintained by Unidata and was last updated May 2013. Send comments to THREDDS support.