UnidataTHREDDS Data Server (TDS) FAQ

General

Catalogs

TDS

Caching

TDS Install Errors

Tomcat

General

Q: I have a strange problem and I need help figuring out whats going on.

A: Here is what we need from you in order to deal with hard problems:

  1. Run the latest stable release. Sorry, we dont have the resources to keep older versions running, so we need to deal with just the latest code.
  2. Get a clean set of logs that capture the problem:
    1. Stop the tomcat server.
    2. Install the latest release if needed.
    3. Remove/copy all files from {tomcat}/logs and {tomcat}/content/thredds/logs
    4. Restart the server
    5. Make the problem happen
    6. Zip up everything in {tomcat}/logs and {tomcat}/content/thredds/logs.
    7. Send the logs and a detailed description of what you do to make the problem happen, and what the problem looks like. If it took a while to get the problem to happen, note what time it happened so that we can correlate with the logs.

Catalogs

Q: How do I construct the URLs I find in a THREDDS Catalog?

A: Heres the general idea in the tutorial and the reference docs. If you are using the CDM library, you can call 

InvAccess.getUrlPath()

Q: How do I eliminate the extra dataset when using a Catalog Reference?

Make the name of the catalogRef the same as the "top" dataset in the referenced catalog. In the following example the name is ""New Point Data".

In the referencing catalog:

 <catalogRef xlink:href="idd/newPointObs.xml" xlink:title="New Point Data" name="" />

In the referenced catalog:

<catalog xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0" 
 xmlns:xlink="http://www.w3.org/1999/xlink" name="THREDDS-IDD OPeNDAP Data Server - New Station Data"
 version="1.0.3">

  <service name="ncdods" serviceType="OPENDAP" base="/thredds/dodsC/"/>
  <dataset name="New Point Data">
    ...
  </dataset>
</catalog>

TDS

Q: How do I keep client requests from overwhelming my server?

There is currently no per-client resource throttle, unfortunately, but we are aware of the eventual need for that. Any given request is single-threaded, so cant hog too many resources. One can limit the size of opendap responses, which tends to be the main problem on our server anyway. See the OPeNDAP section of the threddsConfig.xml page for details.

Q: Can aggregations of many files cause "too many files open" problems?

Aggregations only open one file at a time, and then close it., so this wont cause "too many file" problems.

If you have "too many open files" errors, then either theres a file leak (which we would like to know about), or you have your file cache limit set too high relative to your OS file handle limit.

Q: What do the non-HTTP status codes in the threddsServlet.log files mean?

The "Request Completed" messages in the threddsServlet.log files contain several fields including a status code. the HTTP status code returned in a completed response. If a request is forwarded to another internal service, a "1000 (Forwarded)" or "1001 ( 

2009-06-17T13:25:54.451 -0600 [     28949][      11] INFO
  - thredds.server.catalogservice.LocalCatalogServiceController
  - handlePublicDocumentRequest(): Request Completed - 1001 - -1 - 32

Q: I'm seeing the error "Inconsistent array length read: 538976288 != 1668244581' when I open the dataset in the IDV. Why?

The error "Inconsistent array length read" only tells you that there was an error on the server in the middle of responding to an OPeNDAP request. You then must look in the threddsServlet.log and find the error to know why.

Q: Why am I getting lots of java.util.prefs.BackingStoreException warning messages?

If you allow and use the TDS WMS service, you may be seeing warning messages in your Tomcat catalina.out log file that look something like this:

May 25, 2010 6:28:22 PM java.util.prefs.FileSystemPreferences syncWorld
WARNING: Couldn't flush system prefs: java.util.prefs.BackingStoreException: /etc/.java/.systemPrefs/org create failed.

You can get rid of these messages by setting the "java.util.prefs.systemRoot" system property to a location that is writable by the user that Tomcat runs under.

Here is what we do on our servers:

If you are interested in more details of the problem, here are two useful links:

We have this TDS issue in our bug tracking system and plan to address it in TDS 4.3.

Q: My TDS server is behind a proxy server. Why do some TDS generated URLs point to my TDS server instead of my proxy server?

Most TDS generated URLs are relative to the server (e.g., "/thredds/dodsC/") or relative to the the current document's base URL. There are only a few places where it is necessary to generate absolute URLs. In those cases, the TDS uses information from the incoming HTTP request to construct the generated URLs. It is up to the proxy to send the correct information to the proxied server so the request information will be correct.

For more information, see our web page on running Tomcat behind a proxy server. It contains links to Tomcat documentation on both mod_proxy and mod_jk as well as some user contributed documentation on setting up mod_proxy.

Q: I have modified my configuration of a JoinExisting Aggregation dataset, but nothing has changed.

The files and coordinates in a JoinExisting Aggregations are cached, and in some circumstances wont get updated. The default location for the cache is ${tomcat_home}/content/thredds/cacheAged/ unless you change it in the threddsConfig.xml file. Go to that directory, there will be files with the name of the cached dataset(s). Delete the file for the dataset that needs updating and restart Tomcat.

Q: What happened to the long list of CRSs in my WMS GetCapabilities documents?

In TDS 4.1, each WMS GetCapabilities document listed 100s of available CRS. This made for very large GetCapabilities documents. As of TDS 4.2, this list is limited to a select few CRSs. We hope to make this list configurable in a future release.

In the mean time if you need a CRS that isn't listed, try specifying it in the GetMap request. The underlying library that handles CRS (Geotoolkit) still supports a large number of CRS and the TDS WMS should still support any of those CRS when requested.

Q: Why are TDS web forms not working?

Look in {$TOMCAT}/logs/localhost.logs for error messages like:

     SEVERE: Servlet.service() for servlet GridSubsetService threw exception
   javax.xml.transform.TransformerFactoryConfigurationError: Provider net.sf.saxon.TransformerFactoryImpl not found

If you find these, the likely problem is that another webapp running in the same Tomcat container has set the XSLT parser with javax.xml.transform.TransformerFactory, which is global for the JVM. The above example shows that the Saxon parser has been set, but is not being found by the TDS. We saw this happening with the OOSTethys webapp.

The solution is to move the other webapp to its own Tomcat instance, or to move the required jar (eg saxon.jar) into Tomcat's lib directory, where it is available to all webapps. TDS does very simple XSLT to create its web forms, so its likely that it can work with any decent XSLT library. By default it uses the JDK's built-in XSLT library.

Q: What does the TDS do at startup to read the configuration catalogs? What gets cached? Does it have a way to know a referenced catalog is unchanged? When do referenced catalogs get scanned?

The TDS reads in all the config catalogs at startup. It caches all of them, and uses the "expires" attribute on the catalog to decide if/when it needs to reread a catalog.  It must read all catalogs, including catalogRefs, because it has to know what the possible dataset URLs are, and there is no contract that a client has to read a catalog first.

Q: Why do I get a message that "class org.apache.catalina.ant.DeployTask cannot be found" when I try to build the TDS from source using Ant?

The TDS build.xml file uses several external Ant tasks defined in the catalina-ant.jar file. For TDS 4.2.5 and before, you need to have the catalina-ant.jar file in the lib/ directory of your Ant installation. If it is not there you will get a message that looks something like: 

../tds/build.xml:730: taskdef class org.apache.catalina.ant.DeployTask cannot be found

You can find the catalina-ant.jar file in the lib/ directory of a Tomcat distribution.

For TDS 4.2.6 and above, you should not see this problem as the build.xml file directly references a copy of the catalina-ant.jar file.

Caching

Q: We use compressed netcdf files and the very first access to them are quite slow, although subsequent accesses are much faster, then become slow again after a while. I can see that TDS uncompress these files to the cdm cache directory, but then they must get deleted. Is there a way to keep them in the cache permanently?

Esentially this is a tradeoff between storage spaceand the time to decompress . I assume you dont want to store the files uncompressed, so you have to pay the price of that. To control how these files are cached, see:

http://www.unidata.ucar.edu/projects/THREDDS/tech/tds4.2/reference/ThreddsConfigXMLFile.html#DiskCache

I would suggest that you use:

 <DiskCache>
    <alwaysUse>true</alwaysUse>
    <scour>1 hour</scour>
    <maxSize>10 Gb</maxSize>
  </DiskCache>

and choose maxSize carefully. The default directory is {tomcat}/content/thredds/cache/cdm/ by default, or set it in the above xml.

The trick is to make maxSize big enough to keep the "working set" uncompressed, ie if there is a reletively small "hot" set of files that get accessed a lot, you want to give enough cache space to keep them uncompressed in the cache. Monitor the cache directory closely to see what files stay uncompressed, and how old they are, and modify maxSize as needed.

TDS Install Errors

Q: Why does the TDS fail to deploy in the Tomcat provided by RedHat 5 or 6?

The Tomcat installation provided with RedHat 5 and 6 (and with yum) is run from /usr/share/tomcat5 which contains symbolic links that point into /var/lib/tomcat5. The permissions are such that the TDS cannot write the content/thredds directory in the standard location, ${TOMCAT_HOME}/content/thredds (in this case /usr/share/tomcat5/content/thredds). The error message in catalina.out looks something like this: 

Jun 17, 2009 3:44:08 PM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deploying web application archive thredds.war
log4j:WARN No appenders could be found for logger (org.apache.commons.digester.Digester.sax).
log4j:WARN Please initialize the log4j system properly.
TdsConfigContextListener.contextInitialized(): start.
Jun 17, 2009 3:44:11 PM org.apache.catalina.core.StandardContext start
SEVERE: Error listenerStart
Jun 17, 2009 3:44:11 PM org.apache.catalina.core.StandardContext start
SEVERE: Context [/thredds] startup failed due to previous errors

And there should be a message in the localhost.*.log file that looks something like this: 

ERROR - TdsContext.init(): Content directory does not exist and could not be created [/usr/share/tomcat5/content]

There are currently two solutions:

  1. Create the needed directories by hand and setup appropriate symlinks
    • create the TDS content directory in /var/lib/tomcat5;
    • make sure the tomcat user owns the content directory and has read/write permission; and
    • create the symlink /usr/share/tomcat5/content that points to the directory created above.
  2. Give the TDS an absolute path to the desired location for the content directory by setting the tds.content.root.path system property with a command-line argument (e.g., "-Dtds.content.root.path=/some/absolute/path"). More information is available here.

Q: ERROR - TdsContext.init(): Content directory does not exist and could not be created

The TDS needs to create the directory ${tomcat_home}/content but it does not have permission, typically because ${tomcat_home} is owned by root. Make ${tomcat_home} owned by the tomcat user, or manually create ${tomcat_home}/content and make it owned by the tomcat user.

Q: On starting up TDS, I get the error "SEVERE: Error listenerStart" and "SEVERE: Context [/thredds] startup failed due to previous errors", and TDS wont start.

where:

  1. Harmless log4j warnings. Someday we'll figure out how to get rid of it.
  2. Various initialization info messages
  3. This is the problem, but it doesnt actually contain enough information to know whats going on. It usually means theres an error in how you set up Tomcat.

Q: What does this error mean: "log4j:ERROR Attempted to append to closed appender named [foobar]" ?

The log4j.xml file has 2 loggers with the same name, that uses the appender "foobar". You must delete one of the loggers.

Tomcat

 

Q: Im getting the error "java.lang.OutOfMemoryError: PermGen space". Whats up?

If you reload the thredds.war webapp enough times without restarting Tomcat, you will eventually run into "java.lang.OutOfMemoryError: PermGen space". This is a known bug in JDK/Tomcat. The only thing to do is to stop and restart Tomcat.

You can increase PermGen using this JVM option:

-XX:MaxPermSize=180m

On the JDK1.6 -server JVM, the default seems to be 64m. However, with enough redeploys , you will eventually run out of PermGen space no matter what your MaxPermSize setting is. We have gotten into the habit of restarting tomcat on our production server whenever we redeploy. Lots of redeploys only happen on our test server.

Resources:

Q: During shutdown I'm getting messages about threads (ThreadLocal) having to be shut down to prevent memory leaks. Whats up?

Tomcat memory leak detection code started logging these messages as of Tomcat 6.0.24. From various posts (see Spring Forum: "ThreadLocal forcefully removed" comment #3 e.g.) it appears that these messages are not a problem but instead a matter of Tomcat finding these objects before they get garbage collected.

Here are a number of related links:

NOTE: We will monitor the status of this Tomcat issue. For now, we do not consider this a TDS bug and will not be working to fix this issue in TDS.

Q: Who is accessing my server?

When you examine the TDS access logs, you can see who is accessing the TDS by IP address. Use nslookup <ip address> to find out the host name.

Q: How can I control whether I want Web crawlers to access my server?

Well-behaved web crawlers are supposed to look for a robots.txt file on the server and follow its instructions. To set up a robots.txt file that excludes web crawlers from crawling your server, follow these directions.

Q: How can I prevent someone from accessing my server?

If your server is being overwhelmed by requests from a particular user/computer, it is best to exclude them using their IP address rather than their hostname (this avoids having to perform a DNS lookup for each request). To do so, edit the ${tomcat_home}/conf/server.xml file and find the "localhost" Host element. Add a RemoteAddrValve Valve element as follows: 

<Host name="localhost" debug="0" appBase="webapps" .. >
  <Valve className="org.apache.catalina.valves.RemoteAddrValve" deny="18\.83\.0\.150" />
  ...
</Host>

The value of the deny attribute must be one or more (comma delimited) regular expressions each of which will be compared to the remote clients IP addresses. For instance:

deny="18\.83\.0\.150,128\.100\.34\.99,128\.117\.140\..*"

NOTE: You need to restart the server before this will take effect.

Q: How do I remove Servlet Autodeploy?

Its recommendded to remove autodetection of changes while Tomcat is running, for performance reasons. In a production environment, its better to explicitly redeploy the application: 

   <Host name="localhost" appBase="webapps" unpackWARs="true" autoDeploy="false"
     xmlValidation="false" xmlNamespaceAware="false">
   ...
  </Host>

Q: How do I remove port 8009 when using tomcat in standalone mode?

Unless you are using Tomcat with the Apache server, comment out this line in server.xml: 

  <Connector port="8009" enableLookups="false" redirectPort="8443" protocol="AJP/1.3" />

Tomcat Resources

THREDDS This document is maintained by Unidata and was last updated May 2011. Send comments to THREDDS support.