What is the IDD?
Unidata has built a community-oriented network to deliver data over the internet called the Internet Data Distribution system (IDD). The IDD system has been a driving force in the development of the TDS, our aim is to distribute all the IDD data. The data ranges from model data to text reports, radar images, satellite images, etc. Click here to see Unidata's current running TDS/IDD server.
Since the TDS uses catalogs to make the data available, the THREDDS's group has developed a set of catalogs specifically for the IDD data so others could set up their own TDS/IDD server easily. Here is a sample catalog that created the NDFD web page in the Unidata's TDS/IDD server. See if you can find the NDFD catalog on the Unidata's TDS/IDD server. The main criteria of using the TDS/IDD catalogs is that the data has to be in predetermined data directories because all TDS catalogs contain location elements where the data resides on the disk. Don't be alarmed, there are TDS/IDD configurations available later in the document that will set these directories structures in place automatically.
The TDS/IDD server configuration has 4 parts:
- configure the TDS to use the IDD catalogs
- configure the IDD/LDM to make the predetermined data directory structures
- manage IDD datasets
- performance enhancements by GRIB Indexing
At this point, it's assumed one has installed the TDS and it's running correctly. If not, see the installation instructions here. The IDD catalog configuration is easy because it only requires changing the default TDS catalog.xml to the IDD catalog. The IDD catalog references all the other IDD catalogs so one change gets all the other IDD catalogs.
This is the IDD catalog that is included in the TDS distribution.
The detailed instructions are:
Change to location of the TDS catalog.xml
% cd ${tomcat_home}/content/thredds
Save the default catalog (catalog.xml)
% mv catalog.xml catalog.dist
Get the TDS/IDD catalog from the distribution
% cp idd/catalog.xml .One can restart the TDS from the manager page
or
As user tomcat, stop and start tomcat.
% cd ${tomcat_home}/bin
% ./shutdown.sh
% ./startup.shAt this point, it is assumed that one has installed the LDM software and it is receiving data from the IDD. The LDM will receive/manage the data files by using specific configurations and utilities files that are available by downloading the tdsidd.zip file. In the LDM home directory, unzip the tdsidd.zip file.
The specialized LDM pqact files that have been developed by THREDDS's to create the correct directory structures and dataset names are now be in the LDM etc directory. The pqact files are:
pqact.thredds // create directories for model, text type products, plus others
pqact.threddsradar // create directories for level II and III radar data
pqact.threddsconduit // create directories for high resolution model data
Edit the pqact.thredds file:
Comment out the pqact entries that contain the line "PIPE decoders" because they require
the installation of the decoders package. If one wants to receive these files the
decoder package must be installed and the binaries place in the LDM decoders directory.
To comment out an entry, put a # as the first character in all the entry lines.Edit the ldmd.conf file by adding the following 3 lines added around the other exec lines in the file:
exec "pqact -f ANY-CONDUIT-NNEXRAD-CRAFT etc/pqact.thredds" exec "pqact -f NNEXRAD|CRAFT etc/pqact.threddsradar" exec "pqact -f CONDUIT|SPARE etc/pqact.threddsconduit"
Once the LDM has been configured, restart the LDM so the new pqact files take affect.
stopping and starting needs to be done as user ldm % ldmadmin stop % ldmadmin start % ldmadmin watch // used to check the log files
The realtime IDD datasets arrive constantly, usually at the rate of 1-25 per second. If the data was not deleted in a timely manner, the LDM data partition would fill up in a matter of hours/days. A ball park figure for the amount of disk space needed to store the IDD data is 500-750 gigabytes for a 7 day period.
There is a script that runs once a day removing files older than the number of days specified in its configuration file. The script managFilesDirs.pl and its configuration file manageFilesDirs.conf, both were downloaded in the zip file. These files contain documentation in the header explaining their usage. The manageFilesDirs.pl is located in the ~ldm/util directory and it takes about 1-2 hours to manage a large directory structure. The manageFilesDirs.conf file resides in the ~ldm/etc directory and doesn't need any modifications. By entering the manageFilesDirs script in crontab, one can regulate the number of files to retain and to keep the the LDM data directory at constant usage level without worry.
Here is an example LDM crontab entry to run the manageFilesDir.pl:
# files/dirs managing script
30 21 * * * util/manageFilesDirs.pl -f etc/manageFilesDirs.conf > /data/ldm/logs/manageFilesDirs.log 2>/dev/null
Performance Enhancements by Grib Indexing
Since the IDD distributes GRIB files and the TDS requires a Grib Index to exist before it can open the files, there is a script to pre-index the files to speed up the access time. The GribIndexer.pl is used with the configuration GribIndexer.conf file, both were downloaded in the tdsidd.zip file. If the standard TDS/IDD setup configuration was implemented only a LDM crontab entry needs to be made.
Here is an example LDM crontab entry to run the GribIndexer.pl
# GribIndexer script
0,10,20,30,40,50 * * * * util/GribIndexer.pl -f etc/GribIndexer.conf > /data/ldm/logs/GribIndexer.log 2>/dev/null
The GribIndexer.pl runs ever 10 minutes making sure the Grib Indexes are current and it also creates the log file GribIndexer.log for status. GribIndexer.pl locks the directories so if 2 indexers are running at the same time they don't step on each other. It's possible that a lock file can be left intact wrongly, so there is a -c flag that will clear all the locks in the directories.
Non Standard Configurations
The script assumes that tomcat is installed at /opt/tomcat or a -t flag is needed with the tomcat location. Also it assumes that the version of Java is obtained from the unix which command or an -j flag is required with the java location. It also assumes that the configuration file GribIndexer.conf is located in the LDM etc directory or a -f flag is needed with the configuration file location.
Non Standard Sample
% util/GribIndexer.pl -t /local/tomcat -j /opt/jdk1.5.0_08/bin/java -f /home/ldm/etc/GribIndexer.conf [-c]