[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #QAY-408307]: LDM Gap in packet sequence Error



Hi Jim,

> I currently have one LDM running and ingesting multicast feeds from our Novra 
> receivers.  I am currently attempting to set up a second box so we have 
> redundancy should our first one fail. I have set this up on the same version 
> of Centos 7 as my working LDM box and using the same ldm version 6.13.10 (I 
> have also tried the .11 with the same results).  When I start the ldm it 
> slows the response of the computer dramatically yet the load on the box 
> reported in top does not show a dramatic increase...hovers around .59 on the 
> load being reported. I then looked at the system messages log and see the 
> errors below.  From what I can see with ifconfig and netstat -a the NIC is 
> not reporting any errors but  it does not look like ldm is happy. I am 
> getting a huge amount of Gap in sequence errors but I am not sure where to 
> start to figure this out.  Can you point me in the right direction with some 
> suggestions of what might be going on with my computer?
> 
> Thanks so much!
> 
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> process_prod.c:216:process_prod() SDUS83 KJKL 041821 /pN0MJKL !nids/ inserted 
> [cat 99 type 4 ccb 2/0 seq 66194527 size 6005]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> process_prod.c:216:process_prod() SDUS81 KGYX 041820 /pN3KGYX !nids/ inserted 
> [cat 99 type 4 ccb 2/0 seq 66194528 size 6771]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> process_prod.c:216:process_prod() NXUS63 KJKL 041822 /pGSMJKL inserted [cat 
> 101 type 4 ccb 2/0 seq 66194529 size 245]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> process_prod.c:216:process_prod() SDUS83 KJKL 041821 /pN0KJKL !nids/ inserted 
> [cat 99 type 4 ccb 2/0 seq 66194530 size 12962]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> process_prod.c:216:process_prod() SDUS83 KJKL 041821 /pN0HJKL !nids/ inserted 
> [cat 99 type 4 ccb 2/0 seq 66194531 size 19256]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> process_prod.c:216:process_prod() SDUS83 KBIS 041814 /pDODMBX !nids/ inserted 
> [cat 99 type 4 ccb 2/0 seq 66194532 size 8028]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:436:pmStart() Gap in packet sequence: 373319149 to 373319155 
> [skipped 5]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:660:pmStart() OOPS, start of new product [66194535 ] with 
> unfinished product 66194533
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:782:pmStart() Product definition header version 1 pdhlen 16
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:783:pmStart() PDH transfer type 67
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:791:pmStart() header length 52 [pshlen = 36]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:793:pmStart() blocks per record 1 records per block 1
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:795:pmStart() product seqnumber 66194535 block number 0 data 
> block size 4024
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:796:pmStart() product header flag 1
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:797:pmStart() prodspecific data length 0
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:798:pmStart() bytes per record 0
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:799:pmStart() Fragments = 5 category 1
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:806:pmStart() next header offset 0
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:807:pmStart() original seq number 0
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:808:pmStart() receive time 1567621375
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:809:pmStart() transmit time 1567621377
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:810:pmStart() run ID 0
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:811:pmStart() original run id 0

The above is the output you get when a gap occurs and a new product starts 
arriving.

Assuming that the signal-to-noise ration on the Novra is OK, the gaps could be 
due to problems re-assembling its UDP by the computer. Have the parameters 
described at 
<https://www.unidata.ucar.edu/software/ldm/ldm-current/utilities/noaaport/index.html#preinstallation>
 be set to the indicated values?

Our support-email system is about to go down, so if you reply, you might not 
get a response until tomorrow.

> Sep  4 18:22:54 wxengine4 noaaportIngester[21540]: 
> productMaker.c:436:pmStart() Gap in packet sequence: 978657331 to 978657351 
> [skipped 19]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21540]: 
> process_prod.c:216:process_prod() EOAP88 KAKQ 041200 
> !grib2/ncep/15/#255/201909041200F086/SWELL/3 - OSEQ inserted [cat 3 type 4 
> ccb 2/1 seq 25698559 size 1802]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:436:pmStart() Gap in packet sequence: 373319158 to 373319178 
> [skipped 19]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:1223:pmStart() Missing fragment in sequence, last 3/66194535 
> this 11/66194537
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:436:pmStart() Gap in packet sequence: 373319178 to 373319195 
> [skipped 16]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:436:pmStart() Gap in packet sequence: 373319195 to 373319212 
> [skipped 16]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21540]: 
> productMaker.c:436:pmStart() Gap in packet sequence: 978657351 to 978657420 
> [skipped 68]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:436:pmStart() Gap in packet sequence: 373319212 to 373319267 
> [skipped 54]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21540]: 
> productMaker.c:436:pmStart() Gap in packet sequence: 978657420 to 978657426 
> [skipped 5]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:436:pmStart() Gap in packet sequence: 373319267 to 373319275 
> [skipped 7]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21541]: 
> productMaker.c:436:pmStart() Gap in packet sequence: 16039591 to 16039598 
> [skipped 6]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21541]: 
> process_prod.c:216:process_prod() SXXX20 KWAL 041822 inserted [cat 1 type 4 
> ccb 2/0 seq 10224191 size 113]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21541]: 
> process_prod.c:216:process_prod() SXCA60 KWAL 041822 inserted [cat 1 type 4 
> ccb 2/0 seq 10224192 size 115]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21541]: 
> process_prod.c:216:process_prod() SRPA20 KWAL 041822 inserted [cat 1 type 4 
> ccb 2/0 seq 10224193 size 108]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21541]: 
> process_prod.c:216:process_prod() SXMX40 KWAL 041822 inserted [cat 1 type 4 
> ccb 2/0 seq 10224194 size 380]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21541]: 
> process_prod.c:216:process_prod() SXXX50 KWAL 041822 inserted [cat 1 type 4 
> ccb 2/0 seq 10224195 size 409]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21541]: 
> process_prod.c:216:process_prod() SRGA20 KWAL 041822 inserted [cat 1 type 4 
> ccb 2/0 seq 10224196 size 129]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21541]: 
> process_prod.c:216:process_prod() SRNV20 KWAL 041822 inserted [cat 1 type 4 
> ccb 2/0 seq 10224197 size 96]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21541]: 
> process_prod.c:216:process_prod() SRMI20 KWAL 041822 inserted [cat 1 type 4 
> ccb 2/0 seq 10224198 size 120]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21541]: 
> process_prod.c:216:process_prod() SXNE50 KWAL 041822 inserted [cat 1 type 4 
> ccb 2/0 seq 10224199 size 143]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21541]: 
> process_prod.c:216:process_prod() SXWY50 KWAL 041822 inserted [cat 1 type 4 
> ccb 2/0 seq 10224200 size 144]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21541]: 
> process_prod.c:216:process_prod() SXAK50 KWAL 041822 inserted [cat 1 type 4 
> ccb 2/0 seq 10224201 size 212]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21541]: 
> process_prod.c:216:process_prod() SROK30 KWAL 041822 inserted [cat 1 type 4 
> ccb 2/0 seq 10224202 size 130]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21541]: 
> process_prod.c:216:process_prod() SRHI20 KWAL 041822 inserted [cat 1 type 4 
> ccb 2/0 seq 10224203 size 218]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21541]: 
> process_prod.c:216:process_prod() SXWY50 KWAL 041822 inserted [cat 1 type 4 
> ccb 2/0 seq 10224204 size 177]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21541]: 
> productMaker.c:436:pmStart() Gap in packet sequence: 16039611 to 16039630 
> [skipped 18]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21541]: 
> process_prod.c:216:process_prod() SXXX03 KWAL 041822 inserted [cat 1 type 4 
> ccb 2/0 seq 10224223 size 125]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21541]: 
> process_prod.c:216:process_prod() SRTX20 KWAL 041822 inserted [cat 1 type 4 
> ccb 2/0 seq 10224224 size 120]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21541]: 
> process_prod.c:216:process_prod() SRTX20 KWAL 041822 inserted [cat 1 type 4 
> ccb 2/0 seq 10224225 size 144]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21541]: 
> process_prod.c:216:process_prod() SXXX03 KWAL 041822 inserted [cat 1 type 4 
> ccb 2/0 seq 10224226 size 137]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21541]: 
> process_prod.c:216:process_prod() SXXX03 KWAL 041822 inserted [cat 1 type 4 
> ccb 2/0 seq 10224227 size 135]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21541]: 
> process_prod.c:216:process_prod() SXXX03 KWAL 041822 inserted [cat 1 type 4 
> ccb 2/0 seq 10224228 size 158]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21540]: 
> productMaker.c:436:pmStart() Gap in packet sequence: 978657426 to 978657477 
> [skipped 50]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21541]: 
> productMaker.c:436:pmStart() Gap in packet sequence: 16039635 to 16039645 
> [skipped 9]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21541]: 
> process_prod.c:216:process_prod() SXXX50 KWAL 041822 inserted [cat 1 type 4 
> ccb 2/0 seq 10224238 size 279]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21540]: 
> productMaker.c:436:pmStart() Gap in packet sequence: 978657479 to 978657504 
> [skipped 24]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21540]: 
> productMaker.c:436:pmStart() Gap in packet sequence: 978657505 to 978657526 
> [skipped 20]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21540]: 
> productMaker.c:436:pmStart() Gap in packet sequence: 978657526 to 978657542 
> [skipped 15]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21540]: 
> productMaker.c:436:pmStart() Gap in packet sequence: 978657542 to 978657554 
> [skipped 11]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21540]: 
> productMaker.c:436:pmStart() Gap in packet sequence: 978657554 to 978657586 
> [skipped 31]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21540]: 
> productMaker.c:436:pmStart() Gap in packet sequence: 978657586 to 978657589 
> [skipped 2]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:436:pmStart() Gap in packet sequence: 373319275 to 373319309 
> [skipped 33]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21540]: 
> productMaker.c:436:pmStart() Gap in packet sequence: 978657589 to 978657592 
> [skipped 2]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:436:pmStart() Gap in packet sequence: 373319309 to 373319354 
> [skipped 44]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:436:pmStart() Gap in packet sequence: 373319354 to 373319370 
> [skipped 15]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:436:pmStart() Gap in packet sequence: 373319370 to 373319385 
> [skipped 14]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:436:pmStart() Gap in packet sequence: 373319385 to 373319399 
> [skipped 13]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> process_prod.c:216:process_prod() SDUS32 KFFC 041820 /pNMDATL !nids/ inserted 
> [cat 99 type 4 ccb 2/0 seq 66194581 size 165]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:436:pmStart() Gap in packet sequence: 373319400 to 373319430 
> [skipped 29]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:660:pmStart() OOPS, start of new product [66194595 ] with 
> unfinished product 66194582
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:782:pmStart() Product definition header version 1 pdhlen 16
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:783:pmStart() PDH transfer type 71
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:791:pmStart() header length 52 [pshlen = 36]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:793:pmStart() blocks per record 1 records per block 1
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:795:pmStart() product seqnumber 66194595 block number 0 data 
> block size 1842
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:796:pmStart() product header flag 1
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:797:pmStart() prodspecific data length 0
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:798:pmStart() bytes per record 0
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:799:pmStart() Fragments = 0 category 5
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:806:pmStart() next header offset 0
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:807:pmStart() original seq number 0
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:808:pmStart() receive time 1567621376
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:809:pmStart() transmit time 1567621377
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:810:pmStart() run ID 0
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> productMaker.c:811:pmStart() original run id 0
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> process_prod.c:216:process_prod() ISMD16 RUMS 041800 inserted [cat 5 type 4 
> ccb 2/0 seq 66194595 size 1833]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> process_prod.c:216:process_prod() IUPA42 LPMG 041816 inserted [cat 5 type 4 
> ccb 2/0 seq 66194596 size 340]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> process_prod.c:216:process_prod() ISSD01 ESWI 041800 RRA inserted [cat 5 type 
> 4 ccb 2/0 seq 66194597 size 216]
> Sep  4 18:22:54 wxengine4 noaaportIngester[21538]: 
> process_prod.c:216:process_prod() IUPD41 LOWM 041815 inserted [cat 5 type 4 
> ccb 2/0 seq 66194598 size 234]

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: QAY-408307
Department: Support NOAAPORT
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.