Re: [ldm-users] [noaaport] Assistance requested for "Gap in packet sequence" log entries from noaaportIngester

  • To: Gregory Grosshans <gregory.grosshans@xxxxxxxx>
  • Subject: Re: [ldm-users] [noaaport] Assistance requested for "Gap in packet sequence" log entries from noaaportIngester
  • From: Gilbert Sebenste <gilbert@xxxxxxxxxxxxxxxx>
  • Date: Wed, 17 Jun 2020 14:50:10 -0500
Hi Gregg,

It does change the software, so just in case, back up the /home/ldm/etc 
directory, as I don't know how yours is set up.

A carrier to noise of 15 is below the NWS recommended level of 18. Furthermore, 
that signal strength of -51 dBm is pretty low. College of DuPage has theirs at 
-37 dBm, AllisonHouse normally would be around -34 dBm. 

One thing I noticed is that the LNB power is set to "off". Is that correct? If 
you have a Norsat 3120 LNB,
the power should be set to on, with 22 kHz tone on, long line off. Voltage 
range 13 to 18 V.

What also helped a bit was changing the symbol rate to "auto". It does 
fluctuate slightly, and the Novra will track it nicely.

Gilbert

> On Jun 17, 2020, at 2:26 PM, Gregory Grosshans <gregory.grosshans@xxxxxxxx> 
> wrote:
> 
> 
> Hi Gilbert,
> 
> Yes, we use Linux CMCS. The output of the two commands is:
> 
> CMCS 10.0.5.10> show sat
>       Satellite Interface Settings:
> 
>       Receiver MAC Address:   00-06-76-05-00-d8
>       Receiver Mode:          DVBS2
>       Frequency:              1110.0 MHz
>       Symbol Rate:            30.000 Msps
>       ModCod:                 2/3 16PSK
>       Gold code:              0
>       Input Stream Filter:    On
>       Input Stream ID:        18
> 
>       Signal Lock:            On
>       Data Lock:              On
>       Uncorrectable Rate:     0/Second
>       Packet Error Rate:      0.0000e+00
> 
>       Carrier to Noise C/N:   15.0dB
>       Signal Strength:        -51 dBm
> 
> CMCS 10.0.5.10> show lnb
> 
>       LNB Power:              OFF 
> 
> CMCS 10.0.5.10>
> 
> 
> We did follow the pre-installation steps.
> 
> Does the beta version change the software or just have the change to the 
> ldmd.conf to restart the noaaportIngester process?
> 
> Thanks,
> Gregg
> 
> 
> 
>> On Wed, Jun 17, 2020 at 1:23 PM Gilbert Sebenste <gilbert@xxxxxxxxxxxxxxxx> 
>> wrote:
>> Good afternoon Gregg,
>> 
>> Do you use the Linux CMCS as part of the Novra diagnostic toolset? If you do 
>> (and you should!), log in and on the command line, tell me what is the 
>> output of these two commands:
>> 
>> show sat
>> show lnb
>> 
>> Also, did you do the preinstallation steps found in the section at:
>> 
>> https://unidata.ucar.edu/software/ldm/ldm-current/utilities/noaaport/index.html
>> 
>> Ignore the "configuration" section on that page, it needs to be updated, and 
>> I have made UNIDATA aware of that. The logging and performance 
>> considerations section are also correct on that page.
>> 
>> Also, upgrade that LDM using version 6.13.6 to at least 6.13.11. 
>> Additionally, there is a serious NOAAport ingest bug that was fixed in the 
>> 6.13.12.42 beta that crashes the LDM when garbled grib data (caused by a 
>> garbled file or poor SBN reception) is received. College of DuPage, 
>> AllisonHouse, WISC and others have been using it with no issues. it's 
>> available at:
>> 
>> ftp://ftp.unidata.ucar.edu/pub/ldm/beta 
>> 
>> I would recommend having it on just one of the two servers, not both, with 
>> the other having LDM 6.13.11, the only current version UNIDATA officially 
>> supports.
>> 
>> Gilbert
>> 
>>>> On Jun 17, 2020, at 12:34 PM, Gregory Grosshans via noaaport 
>>>> <noaaport@xxxxxxxxxxxxxxxx> wrote:
>>>> 
>>> 
>>> We are replacing legacy SBN ingest software and spinning up the Unidata 
>>> noaaportIngester (i.e. LDM version 6.13.11) on RHEL7 / RHEL6.  
>>> Unfortunately, we continue to receive "Gap in packet sequence" entries in 
>>> various log files (e.g. nwstg, nwstg2, nother, goes and polarsat), and in 
>>> particular the polarsat.log file has many of these entries.  Can you please 
>>> review the information below, ask clarifying questions, and hopefully offer 
>>> suggestions on how to determine the cause of the Gap entries and steps to 
>>> eliminate them?
>>> 
>>> Thank you for your time,
>>> Gregg
>>> 
>>> 
>>> The NOVRA firmware being used is: 
>>> V2R15
>>> 
>>> LDM Version and Server Information:
>>> 6GB tmpfs for LDM (6.13.11) queue on a Dell R410, 64 GB, dual Intel Xeon 
>>> X5667 @3.07GHz, RHEL7.8
>>> 2GB tmpfs for LDM (6.13.6)  queue on a Dell R410, 32 GB, dual Intel Xeon 
>>> X5667 @3.07GHz, RHEL6.10  
>>> 
>>> 
>>> The SBN Dish is a typical NWS SBN dish, feeding a splitter with multiple 
>>> NOVRA boxes on the other side of the splitter.  
>>> 
>>> I've checked with another National Center and they are not receiving these 
>>> Gap entries, and we have tried their noaaportIngester executable on our SPC 
>>> system and still receive the Gap entries.
>>> 
>>> We have tried multiple NOVRA boxes on the RHEL7 server and continue to 
>>> receive Gap entries.  These same NOVRA boxes work on other systems at SPC 
>>> with no issues.  Different NOVRA boxes have been tried with no success 
>>> (i.e. in terms of having no Gap entries in the log files).  
>>> 
>>> There are very infrequent instances of Gaps in packets appearing on other 
>>> systems at the same time, for example on the AWIPS cpsbn1 server in the LDM 
>>> noaaportIngester log file.  Thus indicating the Gap is in multiple systems, 
>>> using different NOVRA boxes and connections, perhaps on the SBN uplink, or 
>>> perhaps downlink if there is local weather (e.g. perhaps LTG) causing 
>>> interference.
>>> 
>>> I've worked with several of my IT colleagues at SPC and we have eliminated 
>>> the possibility of the Gaps in packets as a result of the NOVRA box, and 
>>> connections between pre/post NOVRA box.  This leads us to believe the 
>>> errors are a result of something on the RHEL6 and RHEL7 servers, to 
>>> possibly include the noaaportIntester.
>>> 
>>> noaaportIngester is being invoked via LDM with the following entries from 
>>> ldmd.conf:
>>> 
>>> EXEC    "noaaportIngester -I 10.0.5.50 -m 224.0.1.1  -n -c -u 3 -r 1 -s NMC"
>>> EXEC    "noaaportIngester -I 10.0.5.50 -m 224.0.1.2  -n -c -u 4 -r 1 -s 
>>> GOES -f"
>>> EXEC    "noaaportIngester -I 10.0.5.50 -m 224.0.1.3  -n -c -u 5 -r 1 -s 
>>> NMC2"
>>> EXEC    "noaaportIngester -I 10.0.5.50 -m 224.0.1.4  -n -c -u 6 -r 1 -s 
>>> NOAAPORT_OPT"
>>> EXEC        "noaaportIngester -I 10.0.5.50 -m 224.0.1.5  -n -c -u 7 -r 1 -s 
>>> NMC3"
>>> EXEC        "noaaportIngester -I 10.0.5.50 -m 224.0.1.6  -n -c -u 4 -r 1 -s 
>>> ADD"
>>> EXEC        "noaaportIngester -I 10.0.5.50 -m 224.0.1.7  -n -c -u 7 -r 1 -s 
>>> ENC"
>>> EXEC        "noaaportIngester -I 10.0.5.50 -m 224.0.1.8  -n -c -u 7 -r 1 -s 
>>> EXP"
>>> EXEC        "noaaportIngester -I 10.0.5.50 -m 224.0.1.9  -n -c -u 4 -r 1 -s 
>>> GRW"
>>> EXEC        "noaaportIngester -I 10.0.5.50 -m 224.0.1.10 -n -c -u 4 -r 1 -s 
>>> GRE"
>>> 
>>> 
>>> LDM settings:
>>> 
>>> [ldmcp@sbn1 ~]$ regutil
>>> /delete-info-files : 0
>>> /hostname : sbn1.spc.noaa.gov
>>> /insertion-check-interval : 300
>>> /oess-pathname : /home/ldmcp/etc/OESS-account.yaml
>>> /reconciliation-mode : do nothing
>>> /check-time/enabled : 1
>>> /check-time/limit : 10
>>> /check-time/warn-if-disabled : 1
>>> /check-time/ntpdate/command : /usr/sbin/ntpdate
>>> /check-time/ntpdate/servers : ntp.spc.noaa.gov ntp1.spc.noaa.gov 
>>> ntp2.spc.noaa.gov
>>> /check-time/ntpdate/timeout : 5
>>> /metrics/count : 4
>>> /metrics/file : /home/ldmcp/logs/metrics.txt
>>> /metrics/files : /home/ldmcp/logs/metrics.txt*
>>> /metrics/netstat-command : /bin/netstat -A inet -t -n
>>> /metrics/top-command : /bin/top -b -n 1
>>> /log/count : 7
>>> /log/file : /home/ldmcp/logs/ldmd.log
>>> /log/rotate : 1
>>> /pqsurf/config-path : /home/ldmcp/etc/pqsurf.conf
>>> /pqsurf/datadir-path : /home/ldmcp/data
>>> /scour/config-path : /home/ldmcp/etc/scour.conf
>>> /surf-queue/path : /home/ldmcp/queues/pqsurf.pq
>>> /surf-queue/size : 2M
>>> /server/config-path : /home/ldmcp/etc/ldmd.conf
>>> /server/enable-anti-DOS : TRUE
>>> /server/ip-addr : 0.0.0.0
>>> /server/max-clients : 256
>>> /server/max-latency : 3600
>>> /server/port : 388
>>> /server/time-offset : 3600
>>> /queue/path : /ldmcp/data/queues/ldm.pq
>>> /queue/size : 6000M
>>> /queue/slots : default
>>> /pqact/config-path : /home/ldmcp/etc/pqact.conf
>>> /pqact/datadir-path : /home/ldmcp/data/data
>>> [ldmcp@sbn1 ~]$ 
>>> 
>>> 
>>> Gap entries in various log files starting with new logs starting at ~1537Z 
>>> (from Monday June 15):
>>> 
>>> [ldmcp@sbn1 ~/logs]$ grep Gap *log | more
>>> goes.log:Jun 15 15:37:55 sbn1 noaaportIngester[3457]: 
>>> productMaker.c:439:pmStart() Gap in packet sequence: 513870098 to 514102655 
>>> [skipped 232556]
>>> goes.log:Jun 15 15:38:01 sbn1 noaaportIngester[3456]: 
>>> productMaker.c:439:pmStart() Gap in packet sequence: 551431874 to 551639014 
>>> [skipped 207139]
>>> goes.log:Jun 15 15:38:08 sbn1 noaaportIngester[3449]: 
>>> productMaker.c:439:pmStart() Gap in packet sequence: 72489 to 72516 
>>> [skipped 26]
>>> goes.log:Jun 15 15:38:08 sbn1 noaaportIngester[3453]: 
>>> productMaker.c:439:pmStart() Gap in packet sequence: 72489 to 72516 
>>> [skipped 26]
>>> 
>>> nwstg2.log:Jun 15 15:37:53 sbn1 noaaportIngester[3450]: 
>>> productMaker.c:439:pmStart() Gap in packet sequence: 1469313177 to 
>>> 1469734315 [skipped 421137]
>>> nwstg2.log:Jun 15 15:37:53 sbn1 noaaportIngester[3450]: 
>>> productMaker.c:439:pmStart() Gap in packet sequence: 1469734334 to 
>>> 1469734336 [skipped 1]
>>> nwstg2.log:Jun 15 15:37:53 sbn1 noaaportIngester[3450]: 
>>> productMaker.c:439:pmStart() Gap in packet sequence: 1469734343 to 
>>> 1469734345 [skipped 1]
>>> nwstg2.log:Jun 15 15:37:53 sbn1 noaaportIngester[3450]: 
>>> productMaker.c:439:pmStart() Gap in packet sequence: 1469734353 to 
>>> 1469734355 [skipped 1]
>>> nwstg2.log:Jun 15 15:37:53 sbn1 noaaportIngester[3450]: 
>>> productMaker.c:439:pmStart() Gap in packet sequence: 1469734355 to 
>>> 1469734361 [skipped 5]
>>> nwstg2.log:Jun 15 15:37:53 sbn1 noaaportIngester[3450]: 
>>> productMaker.c:439:pmStart() Gap in packet sequence: 1469734362 to 
>>> 1469734367 [skipped 4]
>>> nwstg2.log:Jun 15 15:37:53 sbn1 noaaportIngester[3450]: 
>>> productMaker.c:439:pmStart() Gap in packet sequence: 1469734367 to 
>>> 1469734377 [skipped 9]
>>> 
>>> nwstg.log:Jun 15 15:37:54 sbn1 noaaportIngester[3448]: 
>>> productMaker.c:439:pmStart() Gap in packet sequence: 560043371 to 560207084 
>>> [skipped 163712]
>>> 
>>> polarsat.log:Jun 15 15:37:56 sbn1 noaaportIngester[3452]: 
>>> productMaker.c:439:pmStart() Gap in packet sequence: 51060086 to 51060113 
>>> [skipped 26]
>>> polarsat.log:Jun 15 15:37:58 sbn1 noaaportIngester[3455]: 
>>> productMaker.c:439:pmStart() Gap in packet sequence: 175690361 to 175761926 
>>> [skipped 71564]
>>> polarsat.log:Jun 15 15:38:07 sbn1 noaaportIngester[3455]: 
>>> productMaker.c:439:pmStart() Gap in packet sequence: 175761926 to 175761990 
>>> [skipped 63]
>>> polarsat.log:Jun 15 15:38:11 sbn1 noaaportIngester[3455]: 
>>> productMaker.c:439:pmStart() Gap in packet sequence: 175761990 to 175761992 
>>> [skipped 1]
>>> polarsat.log:Jun 15 15:38:41 sbn1 noaaportIngester[3455]: 
>>> productMaker.c:439:pmStart() Gap in packet sequence: 175761992 to 175762207 
>>> [skipped 214]
>>> polarsat.log:Jun 15 15:38:44 sbn1 noaaportIngester[3455]: 
>>> productMaker.c:439:pmStart() Gap in packet sequence: 175762207 to 175762269 
>>> [skipped 61]
>>> polarsat.log:Jun 15 15:39:11 sbn1 noaaportIngester[3455]: 
>>> productMaker.c:439:pmStart() Gap in packet sequence: 175762269 to 175762624 
>>> [skipped 354]
>>> polarsat.log:Jun 15 15:39:17 sbn1 noaaportIngester[3455]: 
>>> productMaker.c:439:pmStart() Gap in packet sequence: 175762624 to 175762626 
>>> [skipped 1]
>>> polarsat.log:Jun 15 15:39:47 sbn1 noaaportIngester[3455]: 
>>> productMaker.c:439:pmStart() Gap in packet sequence: 175762626 to 175763154 
>>> [skipped 527]
>>> polarsat.log:Jun 15 15:39:50 sbn1 noaaportIngester[3455]: 
>>> productMaker.c:439:pmStart() Gap in packet sequence: 175763154 to 175763349 
>>> [skipped 194]
>>> polarsat.log:Jun 15 15:39:50 sbn1 noaaportIngester[3455]: 
>>> productMaker.c:439:pmStart() Gap in packet sequence: 175763349 to 175763351 
>>> [skipped 1]
>>> polarsat.log:Jun 15 15:40:17 sbn1 noaaportIngester[3455]: 
>>> productMaker.c:439:pmStart() Gap in packet sequence: 175763351 to 175763945 
>>> [skipped 593]
>>> polarsat.log:Jun 15 15:40:19 sbn1 noaaportIngester[3455]: 
>>> productMaker.c:439:pmStart() Gap in packet sequence: 175763945 to 175763947 
>>> [skipped 1]
>>> polarsat.log:Jun 15 15:40:30 sbn1 noaaportIngester[3455]: 
>>> productMaker.c:439:pmStart() Gap in packet sequence: 175763947 to 175764006 
>>> [skipped 58]
>>> polarsat.log:Jun 15 15:40:30 sbn1 noaaportIngester[3455]: 
>>> productMaker.c:439:pmStart() Gap in packet sequence: 175764006 to 175764037 
>>> [skipped 30]
>>> polarsat.log:Jun 15 15:40:38 sbn1 noaaportIngester[3455]: 
>>> productMaker.c:439:pmStart() Gap in packet sequence: 175764037 to 175764251 
>>> [skipped 213]
>>> polarsat.log:Jun 15 15:40:49 sbn1 noaaportIngester[3455]: 
>>> productMaker.c:439:pmStart() Gap in packet sequence: 175764251 to 175764283 
>>> [skipped 31]
>>> 
>>> ... 2200+ lines of polarsat Gap entries not listed and the last few at 
>>> ~2140Z:
>>> 
>>> Jun 15 21:39:05 sbn1 noaaportIngester[3455]: productMaker.c:439:pmStart() 
>>> Gap in packet sequence: 177780466 to 177780804 [skipped 337]
>>> Jun 15 21:39:18 sbn1 noaaportIngester[3455]: productMaker.c:439:pmStart() 
>>> Gap in packet sequence: 177780804 to 177780838 [skipped 33]
>>> Jun 15 21:39:22 sbn1 noaaportIngester[3455]: productMaker.c:439:pmStart() 
>>> Gap in packet sequence: 177780838 to 177780976 [skipped 137]
>>> Jun 15 21:39:31 sbn1 noaaportIngester[3455]: productMaker.c:439:pmStart() 
>>> Gap in packet sequence: 177780976 to 177781214 [skipped 237]
>>> Jun 15 21:39:31 sbn1 noaaportIngester[3455]: productMaker.c:439:pmStart() 
>>> Gap in packet sequence: 177781214 to 177781260 [skipped 45]
>>> Jun 15 21:39:45 sbn1 noaaportIngester[3455]: productMaker.c:439:pmStart() 
>>> Gap in packet sequence: 177781260 to 177781537 [skipped 276]
>>> Jun 15 21:39:45 sbn1 noaaportIngester[3455]: productMaker.c:439:pmStart() 
>>> Gap in packet sequence: 177781537 to 177781629 [skipped 91]
>>> Jun 15 21:39:52 sbn1 noaaportIngester[3455]: productMaker.c:439:pmStart() 
>>> Gap in packet sequence: 177781630 to 177781632 [skipped 1]
>>> Jun 15 21:40:05 sbn1 noaaportIngester[3455]: productMaker.c:439:pmStart() 
>>> Gap in packet sequence: 177781632 to 177782030 [skipped 397]
>>> Jun 15 21:40:09 sbn1 noaaportIngester[3455]: productMaker.c:439:pmStart() 
>>> Gap in packet sequence: 177782030 to 177783125 [skipped 1094]
>>> Jun 15 21:40:09 sbn1 noaaportIngester[3455]: productMaker.c:439:pmStart() 
>>> Gap in packet sequence: 177783125 to 177783202 [skipped 76]
>>> [ldmcp@sbn1 ~/logs]$ !! |wc
>>> grep Gap po*log | wc
>>>    2254   33810  302849
>>> [ldmcp@sbn1 ~/logs]$ 
>>> 
>>> 
>>> Have you seen Gaps in packet log entries in the past and if so how did you 
>>> mitigate them and what do you suggest to eliminate these extraneous entries?
>>> 
>>> -- 
>>> =====================================================================
>>> Email seems to be generating increasing inefficiencies in organizations.  I 
>>> learned from a manager a Stanford Computer Science professor no longer uses 
>>> email for communication, but uses SNAIL mail, telephone calls, and person 
>>> to person visits.  I'm considering the same.  
>>> 
>>> Storm Prediction Center
>>> 120 David L. Boren Blvd, Suite 2330
>>> Norman, OK 73072
>>> _______________________________________________
>>> NOTE: All exchanges posted to Unidata maintained email lists are
>>> recorded in the Unidata inquiry tracking system and made publicly
>>> available through the web.  Users who post to any of the lists we
>>> maintain are reminded to remove any personal information that they
>>> do not want to be made public.
>>> 
>>> 
>>> noaaport mailing list
>>> noaaport@xxxxxxxxxxxxxxxx
>>> For list information or to unsubscribe, visit: 
>>> https://www.unidata.ucar.edu/mailing_lists/ 
> 
> 
> -- 
> ********************
> I'm primarily telecommuting.  If you need to reach me, call my work cell 
> 405-249-5310.
> ========================================================================
> Email seems to be generating increasing inefficiencies in organizations.  I 
> learned from a manager a Stanford Computer Science professor no longer uses 
> email for communication, but uses SNAIL mail, telephone calls, and person to 
> person visits.  I'm considering the same.  
> 
> 405-325-2462
> Storm Prediction Center
> 120 David L. Boren Blvd, Suite 2330
> Norman, OK 73072
  • 2020 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the ldm-users archives: