NOAAPORT Data Ingestion

Table of Contents


This system captures NOAAPORT broadcast UDP packets from a DVB-S receiver, creates LDM data-products from the data, and inserts those products into an LDM product-queue. The main programs of this system are 1) the noaaportIngester program; 2) the deprecated dvbs_multicast program; 3) a shared-memory FIFO ; and 4) the deprecated readnoaaport program. The shared-memory FIFO is used to buffer the flow of data from the dvbs_multicast program to the readnoaaport program.


Operating systems based on Redhat Linux (including Fedora) must have two kernel parameters modified from their default values. The parameter net.ipv4.ipfrag_max_dist must be modified from its default value of of 64 in order for all broadcast UDP packets to be correctly received without any gaps. Its value should be 4096 or higher or the value should be 0 in order to disable the check of the sequencing of IP fragments. The value of parameter net.ipv4.conf.default.rp_filter should be 2 in order to obtain correct support for a multi-homed system. For example, as root, execute the commands

 sysctl -w net.ipv4.ipfrag_max_dist=4096
 sysctl -w net.ipv4.conf.default.rp_filter=2
 sysctl -p

If, after making these changes and (re)starting the LDM, the LDM doesn't see any incoming data, then you should try rebooting your system.


Edit the LDM configuration file, ~/etc/ldmd.conf, and add the entries needed to read and process the DVB-S stream(s). Here's one possibility using the noaaportIngester program:

 # DVB-S ingest
 EXEC    "noaaportIngester -m"
 EXEC    "noaaportIngester -m"
 EXEC    "noaaportIngester -m"
 EXEC    "noaaportIngester -m"

Alternatively, the deprecated dvbs_multicast and readnoaaport programs could be used:

 # DVB-S broadcast UDP listening and shared-memory writing processes
 EXEC    "dvbs_multicast -m"
 EXEC    "dvbs_multicast -m"
 EXEC    "dvbs_multicast -m"
 EXEC    "dvbs_multicast -m"

 # Shared-memory reading and data-product creation & insertion processes
 EXEC    "readnoaaport -m"
 EXEC    "readnoaaport -m"
 EXEC    "readnoaaport -m"
 EXEC    "readnoaaport -m"

These ldmd.conf actions create a dvbs_multicast process for each available PID from the UDP multicast stream. Each such process places the received packets into a shared-memory buffer for reading by a corresponding readnoaaport process. There must be one readnoaaport process for each dvbs_multicast process.

Restart the LDM system, if appropriate.


The programs in this system use the LDM logging facility. The default log file for the LDM is, typically, ~/logs/ldmd.log. We've found it useful to override the default logging and have each instance of noaaportIngester or readnoaaport write a notice of every processed product to its own log file. We do this via the -u X option, where X refers to the localX logging facility:

 EXEC    "noaaportIngester -m -n -u 3"


 EXEC    "readnoaaport -m -n -u 3"

If you are not interested in logging to seperate files, simply omit the -u X option.

Since LDM logging uses syslogd(8), one must add additional configuration lines to /etc/syslog.conf. The standard additions to /etc/syslog.conf for the LDM are:

Here is an example of how this looks on our Fedora Linux system:

 *.info;mail.none;news.none;authpriv.none;cron.none;local0.none       /var/log/messages

 # LDM logging
 local0.debug                                                         /home/ldm/logs/ldmd.log

To setup syslogd(8) to log to a different file for each ingest script, one has to add more entries to /etc/syslog.conf. Here is an example of how we have /etc/syslog.conf setup on our Fedora system that is running the LDM and ingesting the nwstg2 and oconus streams in the current DVB-S NOAAPORT stream:

 *.info;mail.none;news.none;authpriv.none;cron.none;local0.none;local3.none;local4.none;local5.none;local6.none  /var/log/messages

 # LDM logging
 local0.debug                         /home/ldm/logs/ldmd.log

 # LDM NOAAport ingest logging
 local3.debug                         /data/tmp/nwstg.log
 local4.debug                         /data/tmp/goes.log
 local5.debug                         /data/tmp/nwstg2.log
 local6.debug                         /data/tmp/oconus.log

The log files for our NOAAPORT ingest processes can become very large in a hurry. Because of this, it is best to rotate the files once-per-day. Here is the crontab(1) entry we currently use to do the file rotation:

 # rotate NOAAport ingest logs
 0 0 * * * bin/nplog_rotate 4

Here, 4 is the number of days of all DVB-S readnoaaport invocation log files to keep online.

Performance considerations if you use dvbs_multicast

dvbs_multicast is the program which reads the UDP packets from a PID on the multicast. The reading of the UDP stream must be able to keep up with the stream since there is no mechanism for UDP packets to be retransmitted. We have found that the process which reads these UDP packets can see gaps in the data stream if the process is swapped out by the operating system. To alleviate this possibilty we recommend the following:

 All Files Functions