LDM Reference Glossary and Index

This is a hyperlinked glossary and index of the terms, and phrases used to explain the fundamentals of the Unidata LDM.

A  B  C  D  E  F  G  H  I  K  L  M  N  O  P  R  S  T  U  W V Y

A

ACCEPT entry
An entry in the LDM configuration-file ldmd.conf that allows an upstream LDM to send unrequested data-products. Description of entry in ldmd.conf.
ALLOW entry
An entry in the LDM configuration-file ldmd.conf that allows a downstream LDM to connect and indicates what data-products it is allowed to receive.
anti-DOS feature
A feature of the LDM that mitigates deliberate or accidental denial-of-service attacks by ensuring non-overlapping feeds to every downstream host. This feature is enabled by default but can be disabled through the enable-anti-DOS parameter of the LDM registry. See the description of the behavior of an upstream LDM process for details.
asynchronous
Describes something that does not cause the initiator to wait until the thing is finished. For example, an asynchronous RPC message does not cause the sender to wait for a reply.
autoshift
Describes the capability of a downstream LDM 6 to change the transfer-mode of its connection depending on its success-rate of inserting data-products into the product-queue. LDM 6.4 was the first version to have this capability.

B

BLKDATA
A message from an upstream LDM to a downstream LDM containing data-product data. LDM 6 protocol message.

C

child process
A UNIX process created by a parent process. Official description.
CONDUIT
A feedtype comprising a huge volume of model output.
creation-time, data-product
See data-product creation-time.
crontab utility
A UNIX utility for scheduling the periodic execution of commands. Official description. See suggested entries for the LDM user's crontab(1) file.
COMINGSOON
An RPC message from an upstream LDM to a downstream LDM containing data-product metadata asking if the downstream LDM wants the corresponding data-product. LDM 6 protocol message.

D

data-product
The fundamental quantum of data acted upon by an LDM. Structure of a data-product.
data-product data
The data portion of a data-product. Further information.
data-product creation-time
The time when the data-product was created by the relevant LDM ingester.
data-product identifier
A character-string that uniquely identifies a data-product. Product identifiers are created by LDM ingesters using the policy of the administrative entity responsible for the relevant feedtype.
data-product latency
The difference between when a data-product is received by a downstream LDM and the data-product's creation-time.
data-product metadata
The metadata portion of a data-product. Further information.
data-product origin
A character-string that identifies the computer on which the LDM ingester was running when it created the data-product. Typically, this is the fully-qualified hostname of the computer.
data-product selection-criteria
The criteria used for selecting data-products. The criteria is composed of a start-time and a stop-time that must bracket the data-product creation-time of matching products, and a list of pairs of feedtype and data-product identifier ERE-s.
data-product signature
An array of 16 unsigned bytes that is unique to a data-product. Signatures are computed by LDM ingesters using the MD5 checksum algorithm. An MD5 checksum webpage.
data-product size
The number of bytes in the data section of a data-product. The value must be less than or equal to 0xFFFFFFFF (4294967295).
decoder
A feedtype-specific program that disposes of data-products that it receives from the pqact process. Further information.
deprecated
This item or feature might go away in a future release. Users are cautioned against using or depending on it.
downstream host
The computer on which a downstream LDM is running.
downstream LDM
Typically, a forked, child process of ldmd that sends a FEEDME request to an upstream LDM and, consequently, receives data-products, which it then writes into the product-queue. In atypical situations, this phrase can also refer to a notifyme process.
downstream LDM 5
A downstream LDM that uses the LDM 5 protocol. For backward compatibility, LDM 6 is capable of acting as a downstream LDM 5.
downstream LDM 6
A downstream LDM that uses the LDM 6 protocol. Behavior.

E

EXEC entry
An entry in the LDM configuration-file ldmd.conf that tells the top-level ldmd to asynchronously execute a program as a child process on startup.
environment variable
A generally mutable object with a name and a character-string value that can be set by a user via the shell or by a program. Copies of environment variables are inherited by child processes. Official description.
execvp() function
A UNIX function that executes a command-string. Official description.
ERE
Extended Regular Expression. ERE-s provide a mechanism to select specific strings from a set of character strings. ERE webpage.

F

FEEDME
A message from a downstream LDM to an upstream LDM requesting data-products. LDM 6 protocol message.
feedtype
A class of data-products coming from a common source (e.g., WMO, NPORT). Feedtypes may be hierarchically arranged. A feedtype may be represented as an integral value (e.g., 1) or as a name (e.g., PPS). Feedtype webpage.
fork
A UNIX system-call that duplicates the invoking parent process to create a new child process. Official description.

G

gateway LDM
A site's LDM that receives data from offsite or that sends data to an offsite LDM
gdbm database
A database based on the GNU dbm database routines.
GEMPAK
A UPC-supported package for analyzing and displaying primarily meteorological data (with emphasis on gridded model output). GEMPAK homepage.
gnuplot utility
A freely-available, 3rd-party UNIX utility for graphing data. GNU distribution.
grep utility
The UNIX utility for searching text files. Official description.

H

HEREIS
A message from an upstream LDM to a downstream LDM that contains a data-product.
HIYA
An unsolicited message from an upstream LDM to a downstream LDM specifying what data-products are about to be sent. LDM 6 protocol message.

I

IDD
The Unidata Internet Data Distribution system. The IDD system is a network of LDMs and their associated institutions that cooperate to distribute meteorological and related data-products for research and education under the auspices of the Unidata program. IDD homepage.
IDD rtstats webpages
A set of webpages for displaying the statistics of non-gateway LDMs on the Unidata IDD that are reported via the rtstats utility. Top-level webpage. Use in monitoring the LDM.
identifier, data-product
See data-product identifier.
ingester
A feedtype-specific program that creates data-products and inserts them into the product-queue. Further information.
insertion-time
The time at which a data-product was inserted into the local product-queue.
iostat utility
A commonly-available UNIX utility for displaying system I/O statistics.
IS_ALIVE
A message from a downstream LDM to the top-level ldmd process on an upstream host inquiring whether or not the relevant upstream LDM process is still running. LDM 6 protocol message.

K

kill utility
The UNIX utility for sending signals to processes. Official description.

L

LDM
Local Data Manager. A system of software for the distribution and processing of data-products in near real-time via the Internet. LDM homepage.
LDM, downstream
See downstream LDM.
LDM, receiving
See downstream LDM.
LDM, sending
See upstream LDM.
LDM, upstream
See upstream LDM.
LDM 4
An LDM that uses the LDM 4 protocol.
LDM 4 protocol
The LDM protocol used by version 4 of the LDM.
LDM 5
An LDM that uses the LDM 5 protocol.
LDM 5 protocol
The LDM protocol used by version 5 of the LDM.
LDM 6
An LDM that uses the LDM 6 protocol.
LDM 6 protocol
The LDM protocol used by version 6 of the LDM. LDM 6 protocol webpage.
LDM log file
The file used to log diagnostic messages from LDM programs. Format and interpretation. Rotation and purging.
LDM protocol
The set of RPC messages used by a version of the LDM to communicate with another LDM of the same version.
LDM registry
An LDM-specific database of configuration parameters. LDM registry webpage.
LDM user
The user on a computer system that owns the local LDM installation. Typically, the user's name is "ldm". LDM user webpage. Creating user-account for. Becoming the LDM user.
ldmadmin
Perl script for managing an operational LDM system. Further information.
ldmd
The top-level LDM server.
ldmping
A utility that temporarily connects to an upstream LDM to test its availability.
ldmd.conf
The configuration-file for the top-level ldmd process. ldmd.conf webpage.
lex utility
A UNIX utility for generating a lexical scanner. Official description.
localn
The facility used by the LDM for logging messages via the syslog daemon. By default, the local0 facility is used. Configuring the syslog.conf file. Setting the facility via the configurescript.
logfile, LDM
See LDM logfile.

M

make utility
A UNIX utility for maintaining, updating, and regenerating groups of programs. Official description.
man utility
A UNIX utility for displaying reference-documentation on UNIX utilities. Official description. User-support example.
MANPATH
The environment variable that determines what directories are searched for manual pages and in what order. Setting in preinstallation step. User-support reference. The man(1) utility.
maximum HEREIS size
The threshold, in bytes, above which an upstream LDM 6 uses a COMINGSOON/BLKDATA message-pair to send a data-product rather than a single HEREIS message.
McIDAS
A UPC-supported package for analyzing and displaying primarily meteorological data (with emphasis on satellite imagery). McIDAS homepage.
metadata, data-product
See data-product metadata.
minimum virtual residence time
The minimum, observed virtual residence time since this parameter was initialized (either by creation of the product-queue or via the pqutil utility. If this interval is not greater than the maximum data-product latency configuration parameter, then a downstream LDM might not reject a duplicate incoming data-product.

N

netstat utility
A utility for displaying network connections.
NOAAPORT
A satellite broadcast network run by NOAA to disseminate weather data to field offices. The LDM can be built to ingest data-products from the NOAAPORT broadcast.
NOTIFICATION
A message from an upstream LDM to an downstream LDM containing data-product metadata. LDM 6 protocol message.
NOTIFYME
An RPC message from a downstream LDM to an upstream LDM requesting data-product metadata. LDM 6 protocol message.
notifying upstream LDM 5
An upstream LDM 5 that sends NOTIFICATION messages to a downstream LDM 5 rather than data-products.
notifyme utility
A utility that makes a NOTIFYME request to an upstream LDM and, optionally, writes the data-product metadata that it receives to an LDM logfile.
NTP
The Network Time Protocol. This protocol is used to obtain time information from a time-server. See also ntpd.
ntpd
The Network Time Protocol Daemon. This freely-available program ensures that a computer's system clock is accurate, which is one of the prerequisites for correct operation of the LDM. ntp.org home page.
ntpdate
A utility that uses the Network Time Protocol (NTP) to either check the accuracy of the system clock or to set the system clock. See also ntpd.
NULLPROC
A message used only to flush or verify a connection.

O

origin, data-product
See data-product origin.

P

parent process
The UNIX process that created a child process. Official description.
pathological regular-expression
An extended regular-expression (ERE) that is longer than two characters and has a ".*" prefix.
pathname
A character-string that identifies a file. Pathnames may be absolute or relative to the current working directory. Official description.
perl utility
A freely-available scripting utility. Official webpage.
plotMetrics utility
An LDM utility for plotting time-series of performance metrics. Uses the gnuplot utility.
PIPE action
See pqact PIPE action.
portmapper
A program that maps RPC program numbers and versions to Internet port numbers. The utility rpcbind is a portmapper.
pqact
A program that reads data-products from the product-queue and disposes of them according to user-configurable instructions in a pqact configuration-file. Further information.
pqact EXEC action
An entry in a pqact configuration-file that tells the associated pqact utility to execute the associated program as a child process when a matching data-product is encountered. Configuration-file entry.
pqact PIPE action
An entry in a pqact configuration-file that tells the associated pqact utility to pipe the data-product data from the matching data-product to a program that the pqact utility starts as a child process. Configuration-file entry.
pqact configuration-file
A file that tells the associated pqact utility how to dispose of data-products. Further information.
pqcat
A program that reads data-products from the product-queue and writes them to a file or to standard output. Further information.
pqcheck
A program that checks the writer-counter in the product-queue.
pqing
An LDM ingester that reads meteorological data in WMO or AFOS format from a file or serial port.
pqinsert
An LDM ingester that inserts files.
pqmon
A program that prints a summary of the state of the product-queue. Use in monitoring the product-queue.
pqsend
A program that sends selected data-products from the product-queue to an LDM using HIYA messages and then terminates.
pqsurf
A program that reads data-products that are composite WMO surface bulletins from the product-queue, splits them into individual bulletins, inserts them as data-products into a separate product-queue, and processes the individual bulletins similar to pqact. Deprecated.
pqutil
A utility for interacting with the product-queue.
process
The address space and system resources in which a program executes on a UNIX system. Official description. Child process.
process group
A collection of processes that can be signaled as a group. Official description.
process group leader
The top-level process of a process group. Official description.
product creation-time
See data-product creation-time.
product identifier
See data-product identifier.
product metadata
See data-product metadata.
product origin
See data-product origin.
product-queue
The memory-mapped disk-file used by the LDM system to cache data-products. Typically, data-products are written to the product-queue by LDM ingesters and read from the product-queue by upstream LDM processes.
product signature
See data-product signature.
product size
See data-product size.

R

receiving LDM
See downstream LDM.
RECLASS
An RPC reply that specifies a new data-product selection-criteria. It may be sent from an upstream LDM to a downstream LDM in response to a FEEDME request or from a downstream LDM 5 to an upstream LDM 5 in response to a synchronous LDM 5 HEREIS or COMINGSOON message.
regex utility
A program for validating ERE-s and for testing their efficiency.
registry
See LDM registry.
regutil
A program for managing the contents of the LDM registry. Execute the command man regutil for more information.
REQUEST entry
An entry in the LDM configuration-file, ldmd.conf, that causes a request to be made to an upstream LDM for data-products. ldmd.conf webpage.
residence time
The amount of time a data-product spends in the product-queue before being deleted to make room for more products.
RPC
Remote Procedure Call. A system by which a function-call on one computer is turned into a function-invocation on another computer. The LDM uses a particular implementation called ONC RPC. ONC RPC specification. ONC RPC tutorial.
rpcgen utility
A program for generating RPC-using C code from an input file.
rtstats
A utility that monitors the product-queue and periodically sends summary statistics to a downstream LDM, which is, typically, at the Unidata Program Center. The utility is usually started by the top-level ldmd process due to an entry in the LDM configuration-file, ldmd.conf. IDD rtstats webpages.
runtime symbolic link
A symbolic link in the LDM user's home directory that references (i.e., points to) the version-specific LDM subdirectory that is the current operational one. Removal of. Creation of.

S

scour
A crontab-driven utility for deleting old data-files based on pathname patterns and retention-times.
sending LDM
See upstream LDM.
sequence number
An unsigned, 32-bit number sequentially assigned by the ingester that created the data-product. Ignored by almost everything but existing due to tradition and history.
set-uid-root
If a program is set-uid-root, then it is owned by user "root" and the set-user-ID bit of the file is enabled.
set-user-ID
If a program's set-user-ID bit is on, then, when executed, the user-ID of the resulting process will be that of the owner of the program rather than that of the user who executed the program.
SIGCONT
The UNIX "continue" signal. This signal is used by LDM data-product ingesters and downstream LDMs to notify the LDM process group that a data-product has just been added to the product-queue.
SIGINT
The UNIX interrupt signal. The default action is to terminate a process. Official description.
signature, data-product
See data-product signature.
SIGTERM
The UNIX termination signal. The default action is to terminate a process. Official description.
SIGUSR2
The UNIX, user-defined signal #2. The canonical action taken by an LDM process upon receipt of this signal is to "rotate" the logging-level: notice -> information -> debug -> notice -> ...
size, data-product
See data-product size.
standard error
The diagnostic print-stream of a UNIX process.
standard input
The standard input-stream of a UNIX process.
standard output
The standard print-stream of a UNIX process.
su utility
A non-standard but common utility for becoming another user.
synchronous
Describes something that causes the initiator to wait until the thing is finished. For example, a synchronous RPC message causes the sender to wait for a reply.
syslog daemon
A configurable UNIX daemon that disposes of log messages sent to it by other processes.
system group
A group of UNIX system users. Users within a group will often have write access to the same files and directories. Official description.

T

TCP
Transport Control Protocol. An low-level Internet protocol for reliably transferring a stream of bytes from one computer to another. TCP tutorial (including TCP State Transition Diagram). Original RFC 793.
time utility
A UNIX utility for timing a simple command. Official description.
top utility
A freely-available, 3rd-party UNIX utility for displaying system resources and utilization. GNU distribution.
transfer-mode
One of two modes in which a downstream LDM can receive data-products. In the primary transfer-mode, data-products are received via asynchronous HEREIS messages. In the alternate transfer-mode, data-products are received via synchronous COMINGSOON and BLKDATA messages.

U

uldbutil
A utility for listing and managing the contents of the upstream LDM database. Execute man uldbutil for the manual-page.
Unidata Program Center (UPC)
The UCAR office that coordinates the Unidata program. UPC homepage.
upstream filter
The data-product filter that an upstream LDM 6 uses to decide whether to send a data-product to a downstream LDM 6. The filter has two parts: 1) an "OK" filter that data-products must match in order to be sent; and 2) an optional "not" filter that data-products must not match in order to be sent. Such filters are specified in the ALLOW entries of the LDM configuration-file, ldmd.conf.
upstream host
The computer on which a upstream LDM is running.
upstream LDM
Typically, a forked, child process of ldmd that reads data-products from the product-queue and sends the data-products or the data-product metadata to a downstream LDM as a result of a FEEDME or NOTIFYME request. In atypical situations, this phrase can also refer to a pqsend process.
upstream LDM 5
An upstream LDM that uses the LDM 5 protocol. For backward compatibility, LDM 6 is capable of acting as an upstream LDM 5.
upstream LDM 6
An upstream LDM that uses the LDM 6 protocol. Behavior.
upstream LDM database
A database that contains information on all upstream LDMs running on the local host. The database is transitory: it exists for each LDM session only. It can be accessed via the uldbutil utility.
uptime utility
A freely-available, 3rd-party UNIX utility for displaying system CPU utilization. GNU distribution.
user-account
The account on a UNIX system for a user of the system. Official description.

V

virtual residence time
The time difference between when a data-product is created and when it is removed from the product-queue. For each product, it is the sum of that product's latency plus that product's
vmstat utility
A utility for displaying system metrics.

W

writer-counter
A field in the product-queue whose value is the number of times the product-queue has been opened for writing without an intervening close. Further information.

Y

yacc utility
A UNIX utility for generating a grammatical parser. Official description.