[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[NOAAPORT #ZLV-851048]: LDM misses data on NOAAPORT via satellite

Hi Jim,

re:
> I just logged in to get some numbers for you.  As you recall wxengine3 
> was the slower computer and wxengine4 is the faster 16 core computer.


Thanks for reminding me!

re:
> wxengine3:
> 
> [ldm@wxengine3 logs]$ cat wxengine3_gapcount
> wxengine3:: 20201210.235721: nGap:    194 nFrame:        929 nG1sec:   174 
> nG5sec:   175 nG15sec:   176 nG1min:   177
> wxengine3:: 20201211.235457: nGap:   2985 nFrame:      12799 nG1sec:  2763 
> nG5sec:  2776 nG15sec:  2778 nG1min:  2782
> 
> (we had just started the logging on 12/10 so not an accurate count)
> 
> wxengine4:
> [ldm@wxengine4 logs]$ cat wxengine4_gapcount
> wxengine4:: 20201210.235721: nGap:   3006 nFrame:      12523 nG1sec:  2691 
> nG5sec:  2725 nG15sec:  2760 nG1min:  2786
> wxengine4:: 20201211.235457: nGap:   2969 nFrame:      12225 nG1sec:  2735 
> nG5sec:  2748 nG15sec:  2750 nG1min:  2757
> 
> So, it looks like as you alluded to, they are both having problems, 

Yup, the numbers of Gap messages and associated missed frames logged
yesterday on both of your machines are not good.

re:
> just maybe on different products so I did not notice. 

This is definitely a possibility.

re:
> I believe you said the next step was to compare the actual
> Gap messages to see if it is a one-to-one correlation. I don't
> know if there is a slick way to do this other than just putting
> eyes on it and brute force. I am attaching the gap files for you 
> also.

I have not created a clever way to do the comparisons, sorry.  The
good news is that the BASH scripts that you are now using create
daily files of Gap messages in the ~ldm/var/logs directory.  On
wxengine3, for instance, the file containing all of the Gap messges
would be:

~ldm/var/logs/wxengine3_20201211.gap

I would use a graphical source code comparison utility to display
both files side by side so that they could be checked.  I don't
think that this is really important at this point, however.

re:
> I also have not seen the email from the email list you mentioned, 
> not that my numbers are up to their standard yet.

Apologies!  I added you to the list at the end of our Meet, but
I had a typo that made the address address@hidden instead
of address@hidden ('i' instead of 'j').  Oops !  I just
corrected the typo, so you should start getting the messages
this evening.

Here are the numbers that I sent out yesterday evening:

mistral.srcc.lsu.edu
mistral:: 20201205.180029: nGap: 230866 nFrame:     509038 nG1sec: 214475 
nG5sec: 224669 nG15sec: 228319 nG1min: 230129
mistral:: 20201206.190306: nGap:    539 nFrame:       1272 nG1sec:   133 
nG5sec:   200 nG15sec:   257 nG1min:   353
mistral:: 20201207.222905: nGap:  35592 nFrame:      77657 nG1sec: 24940 
nG5sec: 30761 nG15sec: 33374 nG1min: 34902
mistral:: 20201208.235459: nGap:  53415 nFrame:     118197 nG1sec: 42942 
nG5sec: 48858 nG15sec: 51337 nG1min: 52765
mistral:: 20201209.234628: nGap:   2098 nFrame:       6146 nG1sec:   907 
nG5sec:  1050 nG15sec:  1313 nG1min:  1712
mistral:: 20201210.233309: nGap:    372 nFrame:       1380 nG1sec:   249 
nG5sec:   270 nG15sec:   305 nG1min:   336
mistral:: 20201211.205629: nGap:    405 nFrame:       1496 nG1sec:   221 
nG5sec:   229 nG15sec:   268 nG1min:   321
np1.ssec.wisc.edu
    np1:: 20201205.221930: nGap:     11 nFrame:         12 nG1sec: 4 nG5sec:    
 4 nG15sec:     4 nG1min:     4
    np1:: 20201206.202507: nGap:      3 nFrame:          3 nG1sec: 1 nG5sec:    
 1 nG15sec:     1 nG1min:     1
    np1:: 20201207.160027: nGap:     21 nFrame:         76 nG1sec: 18 nG5sec:   
 18 nG15sec:    18 nG1min:    19
    np1:: 20201208.215807: nGap:      5 nFrame:          9 nG1sec: 1 nG5sec:    
 1 nG15sec:     1 nG1min:     1
    np1:: 20201209.133145: nGap:      2 nFrame:          3 nG1sec: 1 nG5sec:    
 1 nG15sec:     1 nG1min:     1
    np1:: 20201210.200550: nGap:     87 nFrame:        124 nG1sec: 49 nG5sec:   
 70 nG15sec:    77 nG1min:    80
    np1:: 20201211.160458: nGap:     20 nFrame:         53 nG1sec: 18 nG5sec:   
 18 nG15sec:    18 nG1min:    18
np2.ssec.wisc.edu
    np2:: 20201205.235953: nGap:   1949 nFrame:      24432 nG1sec: 69 nG5sec:   
235 nG15sec:   519 nG1min:  1128
    np2:: 20201206.235942: nGap:   1923 nFrame:       2529 nG1sec: 69 nG5sec:   
227 nG15sec:   491 nG1min:  1119
    np2:: 20201207.201507: nGap:   1722 nFrame:   20144742 nG1sec: 516 nG5sec:  
 634 nG15sec:   826 nG1min:  1196
    np2:: 20201208.000000: nGap:      0 nFrame:          0 nG1sec: 0 nG5sec:    
 0 nG15sec:     0 nG1min:     0
    np2:: 20201209.133145: nGap:      1 nFrame:          2 nG1sec: 1 nG5sec:    
 1 nG15sec:     1 nG1min:     1
    np2:: 20201210.151902: nGap:     82 nFrame:         82 nG1sec: 49 nG5sec:   
 70 nG15sec:    77 nG1min:    80
    np2:: 20201211.050049: nGap:     13 nFrame:         37 nG1sec: 13 nG5sec:   
 13 nG15sec:    13 nG1min:    13
leno.unidata.ucar.edu
   leno:: 20201205.183550: nGap:     55 nFrame:        282 nG1sec: 50 nG5sec:   
 50 nG15sec:    50 nG1min:    50
   leno:: 20201206.000000: nGap:      0 nFrame:          0 nG1sec: 0 nG5sec:    
 0 nG15sec:     0 nG1min:     0
   leno:: 20201207.224226: nGap:    103 nFrame:       2932 nG1sec: 97 nG5sec:   
100 nG15sec:   101 nG1min:   102
   leno:: 20201208.024830: nGap:     13 nFrame:         43 nG1sec: 13 nG5sec:   
 13 nG15sec:    13 nG1min:    13
   leno:: 20201209.232814: nGap:     58 nFrame:       1645 nG1sec: 56 nG5sec:   
 57 nG15sec:    57 nG1min:    57
   leno:: 20201210.151902: nGap:    195 nFrame:        580 nG1sec: 161 nG5sec:  
 182 nG15sec:   189 nG1min:   192
   leno:: 20201211.170709: nGap:     44 nFrame:        465 nG1sec: 42 nG5sec:   
 43 nG15sec:    43 nG1min:    43
uni14.unidata.ucar.edu
  uni14:: 20201205.183550: nGap:     59 nFrame:       2681 nG1sec: 49 nG5sec:   
 49 nG15sec:    50 nG1min:    52
  uni14:: 20201206.170424: nGap:      6 nFrame:       2572 nG1sec: 1 nG5sec:    
 1 nG15sec:     1 nG1min:     3
  uni14:: 20201207.224226: nGap:    106 nFrame:       2970 nG1sec: 100 nG5sec:  
 103 nG15sec:   104 nG1min:   105
  uni14:: 20201208.024830: nGap:     14 nFrame:         50 nG1sec: 14 nG5sec:   
 14 nG15sec:    14 nG1min:    14
  uni14:: 20201209.232814: nGap:     62 nFrame:       1684 nG1sec: 59 nG5sec:   
 60 nG15sec:    60 nG1min:    60
  uni14:: 20201210.220201: nGap:    214 nFrame:        645 nG1sec: 175 nG5sec:  
 197 nG15sec:   204 nG1min:   207
  uni14:: 20201211.170709: nGap:     39 nFrame:        395 nG1sec: 38 nG5sec:   
 39 nG15sec:    39 nG1min:    39
awips-ldmcp1.gsd.demonstration.gov
awips-ldmcp1:: 20201205.221246: nGap:       3 nFrame:         26 nG1sec:     1 
nG5sec:     1 nG15sec:    1 nG1min:    1
awips-ldmcp1:: 20201206.073104: nGap:       2 nFrame:         22 nG1sec:     1 
nG5sec:     1 nG15sec:    1 nG1min:    1
awips-ldmcp1:: 20201207.073819: nGap:       4 nFrame:         41 nG1sec:     1 
nG5sec:     1 nG15sec:    1 nG1min:    1
awips-ldmcp1:: 20201208.050603: nGap:       1 nFrame:         20 nG1sec:     1 
nG5sec:     1 nG15sec:    1 nG1min:    1
awips-ldmcp1:: 20201209.150423: nGap:       1 nFrame:          5 nG1sec:     1 
nG5sec:     1 nG15sec:    1 nG1min:    1
awips-ldmcp1:: 20201210.151902: nGap:      82 nFrame:         82 nG1sec:    49 
nG5sec:    70 nG15sec:   77 nG1min:   80
awips-ldmcp1:: 20201211.232255: nGap:       7 nFrame:         17 nG1sec:     2 
nG5sec:     2 nG15sec:    2 nG1min:    2
awips-ldmcp2.gsd.demonstration.gov
awips-ldmcp2:: 20201205.000000: nGap:       0 nFrame:          0 nG1sec:     0 
nG5sec:     0 nG15sec:    0 nG1min:    0
awips-ldmcp2:: 20201206.000000: nGap:       0 nFrame:          0 nG1sec:     0 
nG5sec:     0 nG15sec:    0 nG1min:    0
awips-ldmcp2:: 20201207.000000: nGap:       0 nFrame:          0 nG1sec:     0 
nG5sec:     0 nG15sec:    0 nG1min:    0
awips-ldmcp2:: 20201208.030703: nGap:       1 nFrame:         28 nG1sec:     1 
nG5sec:     1 nG15sec:    1 nG1min:    1
awips-ldmcp2:: 20201209.000000: nGap:       0 nFrame:          0 nG1sec:     0 
nG5sec:     0 nG15sec:    0 nG1min:    0
awips-ldmcp2:: 20201210.151902: nGap:      82 nFrame:         82 nG1sec:    49 
nG5sec:    70 nG15sec:   77 nG1min:   80
awips-ldmcp2:: 20201211.232255: nGap:       9 nFrame:         23 nG1sec:     2 
nG5sec:     2 nG15sec:    2 nG1min:    2
noaaport3.cod.edu
noaaport3:: 20201205.133202: nGap:    590 nFrame:      40122 nG1sec: 571 
nG5sec:   581 nG15sec:   584 nG1min:   586
noaaport3:: 20201206.135026: nGap:     18 nFrame:         18 nG1sec: 15 nG5sec: 
   15 nG15sec:    16 nG1min:    16
noaaport3:: 20201207.180455: nGap:     48 nFrame:        123 nG1sec: 43 nG5sec: 
   43 nG15sec:    43 nG1min:    43
noaaport3:: 20201208.182137: nGap:   1250 nFrame:     127282 nG1sec: 1213 
nG5sec:  1234 nG15sec:  1243 nG1min:  1246
noaaport3:: 20201209.170625: nGap:   1123 nFrame:      89396 nG1sec: 1090 
nG5sec:  1112 nG15sec:  1118 nG1min:  1122
noaaport3:: 20201210.183310: nGap:    779 nFrame:      52560 nG1sec: 712 
nG5sec:   746 nG15sec:   756 nG1min:   761
noaaport3:: 20201211.163614: nGap:     41 nFrame:       4778 nG1sec: 39 nG5sec: 
   40 nG15sec:    41 nG1min:    41
bird01.allisonhouse.com
 bird01:: 20201205.235600: nGap:    156 nFrame:        257 nG1sec: 53 nG5sec:   
 53 nG15sec:    61 nG1min:    76
 bird01:: 20201206.234900: nGap:    126 nFrame:        237 nG1sec: 47 nG5sec:   
 47 nG15sec:    47 nG1min:    52
 bird01:: 20201207.235320: nGap:    143 nFrame:        202 nG1sec: 32 nG5sec:   
 32 nG15sec:    35 nG1min:    41
 bird01:: 20201208.234750: nGap:   6095 nFrame:      61451 nG1sec: 5271 nG5sec: 
 5314 nG15sec:  5648 nG1min:  5903
 bird01:: 20201209.224000: nGap:     97 nFrame:        129 nG1sec: 16 nG5sec:   
 16 nG15sec:    18 nG1min:    21
 bird01:: 20201210.235930: nGap:    208 nFrame:        250 nG1sec: 58 nG5sec:   
 79 nG15sec:    90 nG1min:   102
 bird01:: 20201211.235530: nGap:   4596 nFrame:      55552 nG1sec: 3861 nG5sec: 
 3896 nG15sec:  4147 nG1min:  4383 

So, what to do now?

The source of the errors you are seeing (as measured by Gap messages)
likely being caused by either/both of the following:

- the Carrier to Noise being experienced on your machine is not as good
  as it any of the other sites reporting summary Gap stats

  What we saw was your systems are reporting C/Ns in around 14 dB.
  The lowest C/N being experienced by any of the machines above is
  in the low to mid 15s.

  My gut feeling is that this is _not_ the source of your errors.

- the data path from the Novra S300N that is responsible for the
  non-Weather Wire channels is introducing errors in the UDP streams
  being delivered to wxengine3 and wxengine4

  You will need to get your network folks involved in troubleshooting
  this possibility.

  Reminder: data path problems were _the_ cause of incredibly high numbers
  of Gap messages on the NOAA/GSL machines.  Fixing the data path dropped
  their numbers down to very low levels as you can see above.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: ZLV-851048
Department: Support NOAAPORT
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.