6.13.14 2021-07-27T17:24:00-0600 configure(1): * Defined _XOPEN_SOURCE to be 600 on and only on SunOS systems in order to support the C compiler; otherwise, it's 700 * rpcgen(1): * Require on MacOS system * Use if possible; otherwise, use package-supplied rpcgen(1) output files ldmadmin(1): * Changed match on "ESTABLISHED" in netstat(1) output to "ESTAB" just in case some netstat(1)'s use that instead * Corrected error-message in "addmetrics" command * Added missing "}" to 'while ($_ = $ARGV[0]) {' * Documented the two types of "pqactcheck" invocations in the manual-page * Made more(1) the default pager for "ldmadmin log". The environment variable PAGER will override. * Created "showsettings" command to print information on the LDM installation to assist troubleshooting logging: * Added LOG_LEVEL_FATAL and associated functions/macros * Merged logging levels ALERT, CRIT, & EMERG into just FATAL pqcreate(1): * Fixed manual-page regarding the computation of the default number of product slots * Corrected approximate mean product-size from 51 kB to 140 kB. This value is used when the number of slots registry parameter is "default". pqact(1): * Enhanced log message about pqact(1) being at the oldest end of the queue to include waking up to discover its position in the queue is older than the oldest data-product. The word "oldest" still appears in the warning -- so you can still search on it. * Changed the logging-level associated with the "-log" option of the FILE and STDIOFILE actions from INFO to NOTE to ensure that such messages are always logged regardless of the logging-level of the pqact(1) process * Removed or demoted to DEBUG some redundant ERROR messages related to the inability to pipe to a decoder * Fixed bug in which a pattern/action entry wouldn't have its time of last-use reset if it was already at the head of the most-recently-used list. If this situation persisted for 6 hours, then the entry would be removed from the list and its file descriptor would be closed. This latter situation could only occur if one entry monopolized its pqact(1) process for 6 hours. * Modified the "-strip" option: it no longer deletes characters whose unsigned value is greater than 127. This allows textual products in an extended ASCII character set (e.g., ISO 8859-1). scour(1): * Replaced scour(1) script with multi-threaded C program that scours much faster than the script * Added scour configuration-file to LDM registry * Renamed "scour" script to "scour.sh" in case users still want to use it * Added "-d" option to scour(1) man-page * Increased maximum size of pattern field to MAX_INPUT bytes * Function replacements to accommodate SunOS 5: * getline(3) -> mygetline(3) * fstatat(2) -> lstat(2) rpc(3) library: * Replaced call to bindresvport() in clnttcp_create() with binding to an ephemeral port to eliminate the necessity for having root privileges LDM Registry: * Added SCOUR_EXCLUDE_PATH for pathname of file that lists directories to be excluded by scour(1) Misc: * Removed much lint identified by clang(1) * Removed lint identified by Coverity Scan Documentation: * Removed reference to installing from an RPM because it doesn't exist * Updated the instructions on using metrics to monitor the LDM * Remove obsolete feedtype webpages NOAAPort: * GRIB2 (GEMPAK) Tables: * README: Modified the "clone" command to work-around permission problems * Added: * Ice Thickness (ICETK, (10,2,1,0)) * Rip Current Occurrence Probability (RIPCOP) (10,1,4,0) * Total Water Level Accounting for Tide, Wind and Waves (TWLWAV) (10,3,205,0) * Mean Increase in Water Level due to Waves (SETUP) (10,3,207,0) * Time-varying Increase in Water Level due to Waves (SWASH) (10,3,208,0) * Erosion Occurrence Probability (EROSNP) (10,3,252,0) * dvbs_multicast(1): Replaced non-standard usleep(3) with standard nanosleep(3) * noaaportIngester(1): * Created "-R " option to set the size of the receive buffer in order to reduce the number of missed packets. (The operating system must allow the new size.) * Ensure that the NEXRAD feedtype is assigned to every product whose parameter "product-specific type" in the product-specific header identifies it as being NEXRAD Multicast: * fmtpRecvv3(3): * Corrected logic bug in tracking last multicast product index * Ignore FMTP PDU if header flag is unknown * Log closing of retransmission socket and ensure set to -1 on error * Made handling of retransmitted duplicate BOP more robust: * retxBOPHandler() and BOPHandler(): Made return-value indicate if BOP was invalid or a duplicate * retxHandler(): * Modified to ignore an invalid BOP * No longer relies on a positive product-size to indicate a tracker-map entry * Exception throwing on no tracker-map entry replaced with logging a notice * Created McastProdPar which encapsulates multicast product-index and sequence number object under mutex protection * Improved log message, documentation, and internal commentary * Improved handling of last product whose BOP needn't be requested * fmtpRecvv3.cpp: Replaced with to get PRIu32 macro * Improved debug messages * UdpRecv.cpp: * Corrected reversed "expected" and "actual" values in error message about a too-small payload message * IperfFmtpPdu(1): Created to create invalid FMTP packet for malicious testing by iperf(1) * mldm_sender(1): Removed lint * recv_test(1): Corrected documentation & simplified diagnostic statement * configure.ac & FmtpConfig.h.in: Added _XOPEN_SOURCE * FmtpConfig.h{,.in}: Removed from repository * UdpSend(3): Created private BlackHat class for sending invalid packets based on the environment variable FMTP_INVALID_PACKET_RATIO * Created three levels of message authentication of multicast packets dependent on the value of environment variable FMTP_MAC_LEVEL: - unset or "0": No MAC - "1": SHA256 HMAC - "2": Ed25519 * Up7Down7_test: Increased sleep after product insertions from 1 s to 2 s to give Ed25519 more time * Improved FMTP packet structures regarding I/O handling and padding: * Deleted RETX_REQ_LEN * Replaced flexible arrays with [1] to accommodate g++ 8.3 * Improved BOPMsg structure * Eliminated use of sizeof(struct ...) where appropriate * Converted FMTP PDU size from a compile-time to a runtime parameter DevOps: * Added "regpar.tab" to configured files * Changed "388" to "@LDM_PORT@" in registry * publish(1): * Eliminated deletion of tarballs with the same major and minor numbers but lower bug-fix numbers because some sites -- rather than implementing their own repository -- depend on older versions always being accessible upstream. * Improved messages * .travis.yml: * Enabled "branch_pattern: devel" * push(1): * Tag the commit if and only if it's on the main branch * Execute autoreconf(1) after setting a new version identifier in "configure.ac" * Doxygen: * Doxyfile.developer: Set output directory to "doxygen-output" * Delete obsolete parameters * .git/hooks: Created pre-commit(1) & pre-push(1) * UXF diagrams: Improved and added to repository * dev-makefile(5): Created * check(1): Created * scan(1): Created * configure.ac: * Improved setting of CFLAGS, CXXFLAGS, AM_CFLAGS, & AM_CXXFLAGS * Removed "pqcreate.1" and "regpar.tab" from AC_CONFIG_FILES * deployAll(1): * Added version argument * Modified to use an associative array and to print asynchronous logs at the end * scour/: * testCscour.py(1): Tests scour(1) correctness * aSpringTree(1): Builds a tree of depth and breadth for testing scour(1) * benchMarkIt.sh(1): Tests performance of scour(1) * .gitignore: Deleted inappropriate slash prefix of "scour.conf" * pq/Makefile.am: Moved HAVE_CUNIT to before test_PROGRAMS to enable "make check" on systems with no CUnit package installed * Makefile.am: Added CLEANFILES, DISTCLEANFILES, and MAINTAINERCLEANFILES * pqact/Makefile.am: Decreased time for "make check" 6.13.13 2020-11-03T15:22:01-0700 ldmadmin(1): Switched the order of the bin/ and util/ directories when setting PATH because one site had a script named "util/ldmd". ldmd(1): * Increased the size of the listen(2) queue from 32 to 1024 in order to support the Unidata LDM that's accepting rtstats(1) messages NOAAPort: * GEMPAK Tables: Added new WMO parameters: 000 000 000 010 Temperature 000 002 000 006 Wind Direction 000 002 022 002 Wind Speed (gust) 000 006 001 002 Total Cloud Cover 000 006 013 005 Ceiling 010 000 003 002 Sig Hgt Wind Waves and Swell * GEMPAK NSSL parameter table: * Added MRMS parameters that just appeared in NOAAPort. See : 209 002 005 000 Lightning Prob. next 30min Grid % LgtProb30min 0 0.00 209 006 030 000 MultiSensor QPE 01H Pass1 mm MrgdQpe1HP1 0 -1.00 209 006 031 000 MultiSensor QPE 03H Pass1 mm MrgdQpe3HP1 0 -1.00 209 006 032 000 MultiSensor QPE 06H Pass1 mm MrgdQpe6HP1 0 -1.00 209 006 033 000 MultiSensor QPE 12H Pass1 mm MrgdQpe12HP1 0 -1.00 209 006 034 000 MultiSensor QPE 24H Pass1 mm MrgdQpe24HP1 0 -1.00 209 006 035 000 MultiSensor QPE 48H Pass1 mm MrgdQpe48HP1 0 -1.00 209 006 036 000 MultiSensor QPE 72H Pass1 mm MrgdQpe72HP1 0 -1.00 209 006 037 000 MultiSensor QPE 01H Pass2 mm MrgdQpe1HP2 0 -1.00 209 006 038 000 MultiSensor QPE 03H Pass2 mm MrgdQpe3HP2 0 -1.00 209 006 039 000 MultiSensor QPE 06H Pass2 mm MrgdQpe6HP2 0 -1.00 209 006 040 000 MultiSensor QPE 12H Pass2 mm MrgdQpe12HP2 0 -1.00 209 006 041 000 MultiSensor QPE 24H Pass2 mm MrgdQpe24HP2 0 -1.00 209 006 042 000 MultiSensor QPE 48H Pass2 mm MrgdQpe48HP2 0 -1.00 209 006 043 000 MultiSensor QPE 72H Pass2 mm MrgdQpe72HP2 0 -1.00 * Removed 209 002 004 000 Lightning Probability 0-30min % LgtProb30min 0 -9999.00 because it's not in NSSL's table and because it's long-name is too similar to and it's short-name is identical to (209,002,005) above from NSSL. pqact(1): * Hopefully closed a tiny window-of-vulnerability in which a SIGALRM might be generated after a successful write(2) to a (very) slow decoder but before the alarm is dismissed -- leading to the abnormal termination of the process. This has occurred at more than one site. * Set default SIGALRM handing to "ignore" just in case * Documented "-t " option for piping to a decoder * Added decoder command to pbuf_flush() log messages Multicast: * Up7.c: * Made auto-initialize: changed public up7_init() to static init() * ChldCmd.c: * Demoted log messages from ERROR to NOTE because the message "Success" from the child process isn't an error * Improve handling of OpenSSL: * Move library examination from LDM's to FMTP's configure.ac * Create FmtpConfig.h.in * Add declaration of EVP_PKEY_new_raw_private_key() if necessary * fmtpRecvv3: * Made handling of index of last multicast product more rigorous Developer: * publish.sh(1): * Converted to functional form. Needs to be tested. * Removed FTP references when modifying "versions.inc" on website * Top Makefile: * Added targets "push-gempak-tables" & "pull-fmtp" * push(1): * Made agnostic of branch * Replaced use of "git commit ..." with use of "git add" * .travis.yml: * Made Coverity Scan act on all branches * deployAll(1): * Replaced use of internal shell function with calling deploy(1) * Made all LDM7 deployments asynchronous * configure.ac: * Added "-I" and "-L" checks for /opt/openssl and /usr/lib64/openssl11 6.13.12 2020-09-30T10:25:12-0600 Installation: * Added EOF newline to regpar.tab.in to accommodate Solaris' sed(1) ldmadmin(1): * plotmetrics: * Corrected sense of local/remote port connections. They were reversed all this time. * Put $LDMHOME/util at head of PATH to support customization * Changed URL for GEMPAK tables updates from "git@github..." to "https://..." because the former could require a key ldmd(1): * Top-Level Server: * Eliminated superfluous error messages at process termination about the upstream LDM database not existing if the "-n" (do nothing) option is specified. * Replaced log_log_q() with loc_add() & log_flush() in "Child ... exited with status ..." messages. * Combined hostname and dotted-quad "Denying connection from ..." messages into one * Upstream LDM: * Don't terminate previously-executing upstream LDM with a different protocol version number. * one_svc_run(): Replaced use of select() with poll() * Increase service timeout from 60 s to 120 s * Downstream LDM-6: * Replaced use of IS_ALIVE with fixed timeout of 5 minutes * Created vetProduct() and modified down6_hereis() and down6_comingsoon() to use it. * Changed logging level of "too-old" and "unrequested" products from INFO to WARN in order to unconditionally reveal why products aren't being inserted into the product-queue due to a slow connection (because it's being throttled, for example). If this happens to you, then you'll need to work with your network administrator to allow LDM traffic to pass unhindered. NB: This means a cold-started LDM might log some "too-old" warning messages, initially. * Log a warning that the (PRIMARY/ALTERNATE) transfer-mode won't work correctly if there are more than two REQUESTs for the same feed. * Modified the algorithm for determining when to switch between PRIMARY and ALTERNATE transfer-modes to consider whether or not the same feed is being received via multicast. This will only have an effect if multicast reception is enabled -- which isn't by default. pqact(1): * pqact.c: * Simplified decision on whether or not to write a new state-file by using tvIsNone() on the insertion-time of the last, successfully- processed product * Ensure valid time-of-last-insertion before writing state file at exit * Correct parsing/formatting of state file with negative seconds. * Modified "Starting Up" message to log the command-line * Changed log message at process termination from Behind by %g s to either No product was processed or Last product processed was inserted %g s ago to better reflect the meaning * Replaced log_error_q() with log_add() & log_flush(). * filel.c: * Added 6 hour timeout for inactive entries that, nevertheless, had open file descriptors to eliminate the possibility of pqact(1) holding open thousands of files that have been scoured and, consequently, filling up the disk with invisible files. This also improves the time it takes for the LDM to shut down. * Corrected truncation of output file by the FILE action when both "-overwrite" and "-metadata" are specified. Before, the file would be truncated to the size of the data-product even though it now has a metadata header. Thanks to Bryan for spotting and analyzing this. * Made the removal of the least-recently-used action (and the comcomittant closure of its open file descriptor) repeat until the current action can succeed or a failure occurs * Added use of the above if a child process can't be fork(2)ed for a decoder * Improved log messages for when an action with an open file descriptor is deleted from the list of such actions and its descriptor is closed uldb.c: * Removed lint found by Coverity Scan NOAAPort: * Changed pre-installation instructions to recommend disabling UDP packet sequence checking. * Created keep_running(1) script to ensure that NOAAPort ingesters are restarted if they crash. Added examples to NOAAPort documentation. * Updated main page of Doxygen-based documentation. In particular, corrected discussion of UDP packet reassembly and setting of system parameter net.ipv4.ipfrag_max_dist to zero. * Added new parameters to the NCEP GEMPAK tables: * 000 001 228 010 Flat Ice Accumulation (FRAM) kg m**-2 FICEAC 0 -9999.00 * 010 003 204 000 Freezing Spray - FRZSPR 0 -9999.00 * 000 002 231 000 Tropical Wind Direction Deg TPWDIR 0 -9999.00 * 000 002 232 000 Tropical Wind Speed m s-1 TPWSPD 0 -9999.00 * 000 001 234 008 Precipitation Duration hr PCPDUR 0 -9999.00 * 000 002 009 000 Vertical Velocity (Geometric) m s**-1 DZDT 0 -9999.00 * updateGempakTables(1): * Changed the "make install..." to "mv -f *.tbl..." to make it independent of the build mechanism. * NOAAPortIngester(1): * Changed logging-level of messages for first fragment of GOES imagery from NOTE to INFO. * Modified thread-creation to prevent the program from immediately terminating if the RHEL Insights Client security package is running. That package will, by default, prevent the creation of a thread with non-default scheduling -- even if the process is owned by root. The program will now try to create threads with high-priority, real-time scheduling but will create default threads and log a warning if it can't. The default threads will increase the risk of missing NOAAPort packets. * process_prod.c: * Added feed to notice-level log message about inserted product * Makefile.am: * Replaced ingest-verification files with ones containing new log messages Logging: * Library: * Created macro log_log(level, ...) * Replaced minimum field-width in default log messages with canonical column-start positions to reduce line lengths and CPU usage * Improved concurrency: * Unlocked file and mutex if thread cancelled. Irrelevant for LDM6. * Added automatic deletion of message queue when thread exits, so calling log_free() is no longer necessary. * Added use of pthread_atfork(). Irrelevant for LDM6. * Added "Terminating logging" message at DEBUG priority * refresh_logging(1): * Replaced "command kill" with "/bin/kill" to ensure correct behavior * Documented internally how log_refresh() will cause the log file to be closed/opened *on the next log message* and not before Product-Queue (pq.c): * Changed return values of pq_sequence() and pq_sequenceLock() * Documented that pq_sequence() and pq_sequenceLock() error codes are negative * Improved error-message in pqe_newDirect() * Added error-message to pqe_discard() * Corrected return status of ctl_rel() on mtof() failure Documentation: * Troubleshooting webpage: * Removed the incorrect value of 4096 for the average product size when computing the number of product-slots to create in a new product-queue. Misc: * Removed lint found by Coverity Scan * ldm_formatCmd() * Created to format a command-line from a command vector * ghostname(): * Improved logic in for determining the fully-qualified name of the local host, which is used by rtstats(1). The LDM registry is still checked first, then gethostname(), and then the default value set by the configure script. If, after all this, the name still isn't fully qualified, then a system call will be made in an attempt to make it fully qualified. This means that the "hostname" parameter in the LDM registry needn't be set unless ghostname() doesn't return a fully-qualified hostname or a different name is desired. * ldmfork(3): * Documented `errno` values * Improved internal commentary Multicast: * ChildCmd.c: * Ensured that all pipe(2) file descriptors in the child process are greater than 2 to avoid conflicts when duplicating them to the standard file descriptors (i.e., 0, 1, and 2). * Product-Queue (pq.c): * Disable thread-cancellation on entry and revert on return * pq_sequenceHelper(): * Print number of locked products * The product will now be unlocked upon return if * The offset argument is NULL * The product doesn't match the class specification * An error occurs, either in the product-queue or the caller's function * Added missing decrease of lock count * OESS scripts: * Modified to handle different virtual circuits that, nevertheles, have the same switch name, port name, or VLAN tag * Corrected URL for OESS services * up7.c: * Simplified and improved robustness by eliminating second server-side transport. * Cleaned-up subscription-request logic. Ensured that an error message is logged if anything goes wrong. * request_product_7_svc(): * Added call to svcerr_systemerr() if there's no current subscription * Added test_connection_7_svc() from Down7 * Improved handling of client destruction when subscribing LDM7 disconnects * Added product-index to "Sending missed product" log message * Squashed SIGSEGV when getting number of product-queue slots for product-index map by always opening product-queue * Corrected bug in determining PIM-file parent directory * Replaced use of global product-queue with one given to up7_init() * Made up7_init() and up7_destroy() public * UpMcastMgr.c: * Added check of logging destination for mldm_sender(1) execution * Assert that the logging module doesn't hold a mutex lock before calling fork() * Stop multicast LDM sender, mldm_sender(1), during multipoint VLAN reconfiguration * down7.c: * Increased sleep duration between subscription attempts from 30 seconds to 120 seconds * Increased RPC subscription timeout to 10 minutes * Specified absolute pathname of vlanUtil(1) to workaround Ubuntu's non-standard execvp(2) * Replaced reliance on getting the RPC timeout from the client handle to explicitly setting the subscription-timeout because the RPC timeout is unset until the first call to `clnt_call()` * Included RECEIVE entry in computation of count of number of upstream suppliers of a given feed * Added mutex loccking/unlocking in down7_init() to silence Coverity Scan * Moved test_connection_7_svc to Up7 * Added periodic heartbeat to upstream LDM7 * Improved some log messages * MldmRcvr.c: * eop_func(): Improved INFO message * missed_prod_func(): Added flushing of errors * ProdNotifier.cpp: * startProd(): Improved error-message * missedProd(): Added error-message * MldmRcvrMemory.c: * Removed lint found by Coverity Scan * Replaced searching of YAML mapping with streaming * Improved commentary * FMTP: * Got multicast message authentication working * Modified logMsg(std::string) to always log at NOTE level * fmtpSendv3: * Improved debug logging * Added release of products held by a unicast socket that just disconnected to prevent all products in the queue from eventually being locked * Release products held by socket that just disconnected * Eliminated deadlock * Added rcvrCount() * Added log_info() for duplicate BOP * Initialized numRetrans to -1 * TcpSend: * Explicitly enabled SO_REUSEADDR and SO_KEEPALIVE * fmtpRecvv3: * Moved direct sending of product-EOP to sender on unicast reception thread to adding message to retransmission queue on that thread to obviate deadlock between that thread and sender's retransmission thread. * Improved handling of missed multicast packets * Adapted to use UdpRecv * Modified to only consider valid FMTP messages * OffsetMap: * Added missing deletion of entry by om_get() * Modified OffsetMap::get() to indicate no-entry instead of throwing an exception * UdpRecv.c: * Created * mldm_sender(1): * Improved releasing of product in product-queue: * Added product release on multicast failure * Added debug message about releasing locked product * Added registry parameter "/fmtp-retx-timeout" for FMTP timout in seconds * Modified to use registry parameter "/fmtp-retx-timeout" * Modified "-r " option to be in seconds * Merged mls_tryMulticast() into mls_startMulticasting() * Added error message on fmtpSender_send() failure * Don't multicast product if no FMTP receivers * Modified mcastProd() to return LDM7_NORCVR and not explicitly release product * Eliminated double free()ing of `mcastInfo` * Continue if no receivers * Log more errors * Demoted "Product not multicast because no receivers from INFO to DEBUG * Added notice-level log message (with status) about terminating normally * Up7Down7_test(1): * Added (commented out) single, small product possibility for debugging * vlanUtil(1): * Replaced use of gawk(1)-only bit operations with shell's for portability * Corrected setting of broadcast address string * setMcastRate(1): * Created to set the multicast sending rate. Uses tc(1). Must be executed by root. Developer: * Removed publishing rules from makefiles in favor of using "publish.sh" * send_recv_test.h: * Improved commentary on multicast addresses * configure.ac(1): * Added LDM_BIN_DIR * push(1): * Replaced "commit -a" with list of modified files * Replaced "git push" arguments with "--mirror" * Instrumented OffsetMap * Only deploy to NWSC node6 * ldm7PreInstall: * Added perl and corrected g++ package-name (gcc-g++) * OffsetMap.cpp: * Demoted get() log message from NOTE to DEBUG * pq.c: * Demoted pq_sequenceHelper() log message from NOTE to DEBUG * Added "delayQueue/" to Git. * Corrected target "push-%" in top-level makefile * Removed Authorizer_test from "make check" because GTest segfaults at exit. * pthread_atfork_test.c: * Created * one_svc_run(): * Converted timeout argument from unsigned to int to support indefinite timeout * Down7::up7Proxy_heartbeat(): * Fixed setting of `lastRequest` * send_test(1) & recv_test(1): * Added "-g " option * Made independent of LDM library * Changed HELLO_GROUP to 234.128.117.1 (unicast prefix based) * Added to distribution (but neither built nor installed) * deployAll(1): Enabled California and Maryland * Disabled "make installcheck" in "mcast_lib/ldm7" on Travis CI * pq_test_insert(1): * Increased inter-product gap to 0.1 s * Moved sleep from beginning to end of insertion-loop * deployAll(1): * Added ldm7@idc-uva.dynes.virginia.edu * Changed deployment to uni15 to deployment to uni17 * Top Makefile.am: * Removed superfluous "muticast-check" target * .travis.yml: * Added cat(1) of top-level Makefile on error * Disabled multicast testing because OpenSSL 1.1 isn't available * ldm7/Makefile.am: * Eliminated hmac_test from testing and distribution * Ignore MldmRcvr.h and MldmRcvrMemory.h because of *.hin * pbuf.c, filel.c: * Commented-out excessive debug log messages 6.13.11 2019-05-06T14:30:46-0600 ldmd(8): * Top-level Server: * Eliminated having the operating-system assign a transitory port number for the LDM server if the server couldn't get its official * A receiving LDM can't connect to the server without knowing the transitory port number; * The LDM server doesn't communicate the transitory port number to the outside world; and * Using a transitory port number prevented the LDM from terminating with an error to indicate a problem * LdmConfFile(3): * EXEC action: * Replaced blocking of SIGUSR1 before execvp() with ignoring it * Added ignoring of SIGUSR2 before execvp() * Eliminated memory-leak found by Coverity Scan * Removed calls to open_on_dev_null_if_closed() to accommodate replacement of log_is_stderr_valid() with log_stderr_is_open(), which returns true if stderr is legitimately redirected to /dev/null * Downstream/Receiving LDM: * DownHelp(3): Eliminated SIGSEGV due to incorrect format for `size_t` variable in error message * Renamed getPreviousProdInfo() to getPrevProdInfo() to shorten log messages * Improved log message(s) resulting from socket disconnection pqact(1): * Made processes created via EXEC entries in pqact(1) configuration-files process group leaders so that they don't receive signals sent to the LDM system's process group. This was done because such processes are unlikely to handle such signals. This feature can be disabled via the configure(1) script's option "--disable-setpgid_exec". * pbuf.c: Corrected format for printing of `ssize_t` variable in error message * Removed SIGALRM from handled signals because pq(1) handles it * Demoted prodAction() log message from INFO to DEBUG and improved it Logging: * Improved concurrency locking for access by multiple threads * Removed mutex-locking from implementations: Locking occurs in log.c only * Eliminated calls to pthread_atfork() * Moved log_...() functions from implementations to log.c * Improved return values and initialization in implementations * log.c: * Refactored * Made functions async-signal-safe: * log_roll_level() * log_refresh() * log_avoid_stderr() * Allowed logging levels from LOG_LEVEL_ERROR to LOG_LEVEL_EMERG to be valid * log2slog(3): * Changed alignment of fields from right to left: * Process * Code location * Added description of SIGUSR1 action * Reintroduced line numbers in code location field * Replaced log_is_stderr_valid() (which returned false if stderr was open on "/dev/null") with log_stderr_is_open() (which returns true in that case) in order to handle legitimate redirection of stderr by the user to "/dev/null" * Ensured that all programs check the return-value of * log_init() * log_set_destination() * Eliminated use of deprecated log_syserr_q() Product-Queue (pq.c): Created pq_getMagic() to obtain the magic number for debugging purposes NOAAPort: noaaportIngester(1): * Ensured initialization of the index of the last received frame if and when the run number changes. This lint was found by Coverity Scan. * ldmProductQueue: Created lpqGetMagic() to obtain magic number of product-queue for debugging purposes * Redirected stderr to /dev/null if it's closed because some GRIB/GEMPAK functions mistakenly write to it and it was open on the product-queue -- causing an assertion failure dvbs_multicast(1): Adapted to use extractDecls(1) script in the top-level directory because it includes a warning Misc: Compiled under "gcc -Wall" and fixed almost all warnings Multicast: * ChildCmd(3) (was ChildCommand(3)): * Shortened log messages by * Renaming ChildCommand(3) to ChildCmd(3) * Renaming childCmd_logStdErr() to childCmd_log() * Improved error-logging * Created childCmd_getCmd() * Corrected sudo() * Added debug message that prints command before execution * ldmd.conf(5) * RECEIVE entry: * Corrected description of * Improved description of * MULTICAST entry: * Eliminated parameters * Multicast interface * time-to-live * down7(3): * Corrected AL2S VLAN endpoint <-> FMTP interface consistency-test * Added "sh" to vlanUtil(1) command because Linux won't execute a setuid script that's owned by root * Increased RPC timeout for subscribe_7() because adding a node to an AL2S multipoint VLAN can take longer than the default RPC client timeout. * Modified to pass CIDR address for virtual interface to vlanUtil(1) * Made down7_halt() async-signal-safe to obviate deadlock if interrupted while logging * Differentiated "backlog" from "backstop" product-insertion messages * Removed lint found by Coverity Scan * Added number of FMTP retransmissions to "Received" log message * OESS Scripts: * Replaced calls to ulogger(1) with sys.stderr.write() because sudo() handles the child process writing to stderr * provision.py: * Replaced writing of circuit-ID to stderr with writing to stdout * Added case where multipoint VLAN already exists * Scripts now expect the name of the OESS workgroup to be in the OESS account file * UpMcastMgr(3): * Changed default extension of OESS account-file from "json" to "yaml" * Added installation of account.py(1) and edit.py(1) * Deleted invocation of Python interpreter because scripts are executable * Removed acquisition of root privileges when calling "provision.py" * Added printing of "provision.py" command-string on failure * Deleted logging of child process that error-exits * Added to mldm_ensureExec() removal of terminated sender from mldm_sender_map(3) * Changed AL2S workgroup name from "UCAR-LDM" to "Virginia" so that Yuanlong could more easily diagnose problems * Up7(3): Demoted "Backlog product sent:..." log-message from NOTE to INFO * vlanUtil(1): * Addd "/sbin" to PATH to get ip(1) * Removed address de-assignment from "destroy" because deleting the virtual-interface deletes the address * Added subnet specification to assignment of address to virtual interface * Eliminated use of subnet specifiction when managing route to sender * Added broadcast address to `ip addr add` * Sender code: * Disallowed multicast interface parameter because the interface must be the same as the FMTP server address * Ensured that the FMTP server address is in in the same subnet as its FMTP clients * Internet(3): Created ::to_string(struct sockaddr&) * mldm_sender(1): * Improved MldmRpcCmd server start-up message * Corrected removal of secret file * CidrAddr(3): * Made cidrAddr_copy() more robust to verify subscription reply * Created cidrAddr_snprintf(), which doesn't allocate memory * FmtpClntAddrs::Impl(3): Corrected vetting of FMTP server address in constructor * FMTP: * TcpRecv::initSocket(): Corrected name in log messages * fmtpRecvv3::Start(): Modified to iterate its attempt to connect to the sending FMTP server because it can take time for the AL2S VLAN to be provisioned. It will attempt every 5 seconds for 2 minutes. * fmtpSendv3(3): * ::RunRetxThread(): Don't terminate process on TcpSend::parserHeader() error * Adapted ::logMsg() to use LDM logging * Eliminated this module terminating the process if an exception occurs * Eliminated slicing of the exception that caused an instance to stop working * fmtpRecvv3(3): Added per-product tracking of number of FMTP data-block retransmissions Developer: * Deployment: * deploy(1): * Added determination of "-r" host-suffix * Added removal of runtime link if LDM wasn't running * Added "-r " option * Handle $sudoUser not being able to create log file for "make root-actions" * Added test for "runtime" link and improved messages * Modified to delete previous installed versions * Added "-v" option to "ldmadmin start" * Changed one beep to two * Disabled X11 forwarding by ssh(1) ("-x" option) because it 1) isn't needed; and 2) fails on some hosts * deployAll(1): * Removed determination of "-r" host-suffix * Added the MAX's host * Set privileged user to "ldm7" for sudo(8) on "fdt-uva" * Added University of Missouri's host * Changed host names to state abbreviations * Sequenced deployment to non-NWSC nodes alphabetically * Up7Down7_test(1): Ensured that CU_ASSERT_*() functions aren't called during teardown * GitHub: Added deletion of temporary branch in makefile rule "push-%" * ldm7PreInstall: * Eliminated the use devtools-2's g++ if on a CentOS7 system * Changed an "install" to "update" * Travis CI (.travis.yml): * Corrected command to download Coverity Scan's certificate-chain * Ensured that last "/"-separated component in "secure" declaration starts with letter * Broke download of Coverity's certificate into multiple lines * Added installation of certificate that's missing in Coverity Scan's certificate-chain * Moved dependent package installations from "before_install:" to "addons:apt:packages:" * configure.ac: Modified to allow "./configure ... CC='gcc -std=c99' CFLAGS=-Wall" * SCM: * Deleted "oess" subtree and renamed "oess-client" subtree to "oess" * Fixed broken "pull-%" rule: made it explicit for "oess" and "fmtp" 6.13.10 2019-03-06T14:17:16-0700 Logging: Applied changes to log2slog(3) module to log2ulog(3) module Developer: Redirected publish.sh(1)'s scp(1) output to /dev/null while copying documentation 6.13.9 2019-03-06T13:30:34-0700 ldmping(1): Made the main loop interruptable. Reconciling signal-handling amongst the various programs apparently caused this one to loop forever. pqact(1): Demoted two log messages about writing a product to a file from NOTICE to INFO because per-product messages should be logged only in verbose logging mode ("-v"). Logging: Unconditionally included the ulog(3) logging module in the LDM library for use by third-party software (e.g., McIDAS) Misc: Replaced in ChildCommand(3) with to comply with the X/Open standard Developer: * Modified "publish.sh" to use "www" or "www-r" by default * Improved messaging by deploy(1) and added an alert * Adapted configure(1) to new directory structure of "mcast_lib/OESS-Client" * Added rationale for using longer subtree merging command in makefile * Made version an option in deploy(1) and added user to @host * Eliminated $version and $hostSuffix from deployAll(1) and added host "frgp" * Moved "Deploying..." message from deployAll(1) to deploy(1) 6.13.8 2019-03-04T08:49:35-0700 Building and Installing: * Made the C++ library reference for building the LDM library conditional on WANT_MULTICAST to obviate the need for a C++ library for a regular build * Added missing mutex source-files to distribution to support configure(1)'s "--with-ulog" option ldmadmin(1): * Changed "ldmadmin start" to only print one message and to wait indefinitely for the LDM server to start * Changed "ldmadmin stop" to only print one message and to wait indefinitely for the LDM server to stop * Added the command "updategempaktables" to update the installed GEMPAK tables from the GitHub repository. Requires a standard installation directory structure. Logging: * Refactored logging to eliminate occasional hanging by a terminating ldmd(1) process * Added to two functions a missing return value found by Coverity Scan NOAAPort Ingestion: * Converted the directory "gempak/tables" into its own GitHub repository so that it can be updated and downloaded independent of the LDM package * Added RHEL and CentOS to the documentation as examples of Red Hat derivatives Multicast: * fmtpRecvv3::joinGroup() * Replaced joining an any-source multicast group with joining a source-specific one because not all networks support any-source multicast * Converted "struct ip_mreq_source mreq" from class member to local variable * Added debug log message for successfully joining multicast group * Added debug log message for successful bind() * Added error message if MldmRpc::Impl::~Impl can't remove the secret file * Added removal of MldmRpc(3) secret files to ldmadmin(1) * Modified downlet_run() to print result of changing address of FMTP server * Refactored oess_init() & oess_destroy() to achieve "const" correctness * Adapted to new "provision.py" and "remove.py" command-arguments * Improved OESS-client markup in README.md * Removed double free()ing in UpMcastMgr(3) found by Coverity Scan * Improved documentation of childCmd_reap() * Corrected documentation of up7::init() * Improved documentationi of umm_subscribe() * Saved remote virtual-circuit endpoint for removal in UpMcastMgr(3) Documentation: Added thread-safety and async-signal safety sections to doxygen-generated documentation Misc: Eliminated small and infrequent memory leak found by Coverity Scan Developer: * Removed directory "mcast_lib/OESS-Client" from AC_CONFIG_SUBDIRS() to eliminate warning about that directory not having a configure(1) script * Made directory "mcast_lib/OESS-Client" into the subtree "oess-client" because it's a separate repository. (The FMTP repository is already a subtree.) * Added "pull-%" and "push-%" to the top-level makefile for pulling-from and pushing-to subtrees * Added scripts "ldm7PreInstall" and "ldm7Install" to distribution * Changed multicast group IP address of send_test(1) and recv_test(1) from 234.127.117.0 to 234.128.117.1 and improved logging. * send_test(1) & recv_test(1): * Made independent of LDM package * Made independent of gcc(1) and eliminated compiler warnings * Changed multicast group address from any-source 234... to source-specific 232... * Created script for deploying to a remote host * Created script for deploying to all permitted remote hosts * Created context/container/component diagrams for LDM 6.13.7 2019-02-01T15:19:00-0700 Building and Installing: * Downgraded standard-compliance from _XOPEN_SOURCE=700 to 600 to accommodate Solaris and eliminated reliance on non-standard GNU C functions. Lots of small changes as a result. * Made binary installation directory writable by LDM user only to obviate execution of malicious binary with root privileges. * Added "-lstdc++" to shared-library creation. * Added explicit defines of some C macro values to "1" to avoid problems with configure(1) on BSD systems. * Added "--with-delayQueue" option to configure(1). * Made configure(1)'s "--enable-debug" option set CFLAGS & CXXFLAGS. * Added OESS-Client module * Created ldm7PreInstall(1) and ldm7Install(1) scripts * Corrected setting of installation directories in "config.h" that are overridden at configure(1) time noaaportIngester(1): * Improved responsiveness to external events: * Improved input FIFO: * Made fifo_close() async-signal-safe for calling by the signal handler * Modified fifo_close() to use shutdown() and close() to improve responsiveness * fifo_getBytes() is now notified if either fifo_close() is called or if an EOF is encountered * Added file descriptor to construction and deleted buffer * Modified signal handler to only use async-signal-safe functions * Modified statistics-reporting thread to use sigwait() and eliminated now-unnecessary `reportStatistics` variables * Incorporated fileReader and multicastReader components into noaaportIngester.c * Eliminated file descriptor from input-reader component * Improved construction of LDM product-identifier: * Enabled creation of a complete LDM product-identifier even if the forecast time-interval of a GRIB2 field can't be decoded due to an unknown PDTN. * Added requirement that the GRIB2 version number not be missing (255) and the local version number be 0 or missing (255) in order to obtain the WMO GEMPAK vertical coordinate table. * Made the decision to use GEMPAK's WMO parameter table more robust by requiring that the Local Table number be 0 or 255 (missing). * Modified how GEMPAK finds a parameter entry: * Replaced linear search with binary search * Allowed use of parameter with same discipline, category, and ID but different parameter definition template number if an exact match doesn't exist. An ERROR-level message will be logged in this case. * Added some error-checking to GRIB2 and GEMPAK libraries * Changed default logging level of noaaportIngester(1) from ERROR to WARN LDM registry and regutil(1): * Simplified log messages about missing registry parameters and changed logging level from error to warning. * Replaced obsolete time-server `timeserver.unidata.ucar.edu` with `pool.ntp.org`. * Added pathname of file that contains OESS account information to LDM registry ldmd(1): * Decomposed server into smaller functions. * Eliminated use of portmapper because * Lots of places consider it a security hole * It slowed things down significantly if it wasn't running * It doesn't appear to be used much * Replaced log_*_q() calls with log_*() uldbutil(1): Reduced number of error messages if the LDM isn't running. ldmadmin(1): * "addmetrics" command: * Corrected space-usage numbers on CentOS * Added non-blocking locking of metrics file to eliminate concurrent writes by multiple "ldmadmin addmetrics" processes when under severe load. * "isrunning" command: * Replaced use of kill(1) with ps(1) because some systems misreported success. * Changed how the PATH environment variable is modified to favor version-specific utilities to accommodate non-standard installations that don't have a version-specific subdirectory. plotMetrics(1) (used by "ldmadmin plotmetrics"): * Added "eval" to pathname expansion to accommodate patterns that contain things like "metrics.txt{,.[1-4]}". * Added "-o