[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 20020915 SJSU Latency



HI Mike, 

Barring aeolus's shut down today, it looks as if data is flowing better.

Is that your take on it as well?

We will need to migrate HDS off of motherlode eventually, it is only a
filing and acting machine, we relay off of thelma, but lets hope to get to
a stable state before we go mucking around...

Comments?

-Jeff
____________________________                  _____________________
Jeff Weber                                    address@hidden
Unidata Support                               PH:303-497-8676 
COMET Case Study Library                      FX:303-497-8690
University Corp for Atmospheric Research      3300 Mitchell Ln
http://www.unidata.ucar.edu/staff/jweber      Boulder,Co 80307-3000
________________________________________      ______________________

On Tue, 17 Sep 2002, Mike Voss wrote:

> Jeff,
> 
> I think you caught Aeolus as it was coming back online or something, it's 
> been mostly down since 05Z and down again it seems. I've been having trouble 
> connecting:
> ----
> rossby:~>ldmping aeolus.ucsd.edu
> Sep 17 16:20:21      State    Elapsed Port   Remote_Host           rpc_stat
> Sep 17 16:20:31 SVC_UNAVAIL   9.997790    0   aeolus.ucsd.edu  
> h_clnt_create(aeolus.ucsd.edu): Timed out while creating connection
> ------
> 
> but to motherlode and sunny89 are fine:
> -----
> rossby:~>ldmping motherlode.ucar.edu
> Sep 17 16:22:18      State    Elapsed Port   Remote_Host           rpc_stat
> Sep 17 16:22:18 RESPONDING   0.073225  388   motherlode.ucar.edu 
> -----
> and
> -------
> rossby:~>ldmping sunny89.atmos.washington.edu
> Sep 17 16:23:17      State    Elapsed Port   Remote_Host           rpc_stat
> Sep 17 16:23:17 RESPONDING   0.085064  388   sunny89.atmos.washington.edu  
> --------
> 
> My failover to Sunny89 is problematic, The HDS from motherlode just flies in 
> on time, but I'm not catching up on IDS...strange:
> --------
> rossby:~>ldmadmin watch -f IDS
> (Type ^D or ^C when finished)
> Sep 17 16:32:37 pqutil:      492 20020917152016.968 IDS|DDPLUS 046  NTUS99 
> KNCF 171519 /pTSTNCF
> Sep 17 16:32:37 pqutil:      492 20020917152017.190 IDS|DDPLUS 051  NTUS96 
> KNCF 171519 /pWTSNCF
> Sep 17 16:32:40 pqutil:     9678 20020917152024.316 IDS|DDPLUS 076  SRUS86 
> KRSA 171519 /pRR8RSA
> Sep 17 16:32:40 pqutil:     1818 20020917152024.345 IDS|DDPLUS 081  FGUS56 
> KPTR 171519 /pRVFCW
> Sep 17 16:32:41 pqutil:      120 20020917152024.456 IDS|DDPLUS 086  SRUS55 
> KPIH 171519 /pRR3PIH
> Sep 17 16:32:41 pqutil:      113 20020917152023.843 IDS|DDPLUS 091  SAUS41 
> KBTV 171519 /pMTRBTV
> Sep 17 16:32:41 pqutil:      272 20020917152028.844 IDS|DDPLUS 101  FTUS46 
> KMTR 171100 AAA /pTAFSFO
> Sep 17 16:32:41 pqutil:       98 20020917152028.873 IDS|DDPLUS 106  NZUS87 
> KMFL 171519 /pKPAMFL
> Sep 17 16:32:42 pqutil:     3003 20020917152029.043 IDS|DDPLUS 111  SXUS55 
> KTFX 171519 /pOSOINF
> Sep 17 16:32:42 pqutil:     5873 20020917152029.098 IDS|DDPLUS 116  SRUS54 
> KTUA 171519 /pRR4TUR
> Sep 17 16:32:42 pqutil:     4007 20020917152029.151 IDS|DDPLUS 121  ASUS45 
> KGTF 171519 /pSWRMT
> Sep 17 16:32:43 pqutil:     2327 20020917152029.220 IDS|DDPLUS 126  SXUS08 
> KAKQ 171518 /pOMRAKQ
> Sep 17 16:32:51 pqutil:       98 20020917152029.318 IDS|DDPLUS 131  NZUS87 
> KALR 171519 /pKPAALR
> Sep 17 16:32:57 pqutil:     1248 20020917152037.515 IDS|DDPLUS 156  SXHW50 
> PHFO 171519 /pOSOHI
> Sep 17 16:32:58 pqutil:      602 20020917152039.400 IDS|DDPLUS 166  SRUS56 
> KOTX 171519 /pRR1GEG
> Sep 17 16:32:59 pqutil:       98 20020917152040.103 IDS|DDPLUS 176  NZUS87 
> KSJT 171519 /pKPASJT
> Sep 17 16:33:00 pqutil:      406 20020917152040.328 IDS|DDPLUS 186  SRUS52 
> KGSP 171519 /pRR3GSP
> Sep 17 16:33:05 pqutil:      287 20020917152048.194 IDS|DDPLUS 206  FTUS44 
> KFWD 171519 AAA /pTAFACT
> Sep 17 16:33:25 pqutil:      492 20020917152101.844 IDS|DDPLUS 342  NTUS96 
> KNCF 171520 /pWTSNCF
> Sep 17 16:33:25 pqutil:      492 20020917152101.913 IDS|DDPLUS 347  NTUS99 
> KNCF 171520 /pTSTNCF
> Sep 17 16:33:26 pqutil:       82 20020917152101.934 IDS|DDPLUS 352  SRUS52 
> KMLB 171520 /pRR2MLB
> Sep 17 16:33:30 pqutil:      842 20020917152106.506 IDS|DDPLUS 412  SXUS51 
> KAKQ 171520 /pOSOAKQ
> Sep 17 16:33:33 pqutil:       90 20020917152108.375 IDS|DDPLUS 472  SRUS55 
> KCYS 171520 /pRR3CYS
> Sep 17 16:33:33 pqutil:      198 20020917152108.521 IDS|DDPLUS 477  FTUS46 
> KMTR 171100 AAA /pTAFOAK
> Sep 17 16:33:35 pqutil:      594 20020917152112.759 IDS|DDPLUS 522  ASUS51 
> KALB 171520 /pSWRQE
> Sep 17 16:33:35 pqutil:     1028 20020917152112.822 IDS|DDPLUS 527  AWUS43 
> KGID 171520 /pAWSGRI
> Sep 17 16:33:35 pqutil:      115 20020917152113.233 IDS|DDPLUS 547  SRUS52 
> KCHS 171520 /pRR3CHS
> Sep 17 16:33:39 pqutil:      124 20020917152119.699 IDS|DDPLUS 612  SXUS43 
> KGLD 171520 /pMTTSYF
> Sep 17 16:33:41 pqutil:     3862 20020917152119.850 IDS|DDPLUS 627  SXUS55 
> KTFX 171520 /pOSOGTF
> Sep 17 16:33:41 pqutil:      463 20020917152119.895 IDS|DDPLUS 632  SRUS56 
> KPDT 171520 /pRR3PDT
> Sep 17 16:33:43 pqutil:     3349 20020917152122.900 IDS|DDPLUS 682  NZUS01 
> KBTV 171520 /pSWRNOW
> Sep 17 16:33:54 pqutil:      121 20020917152128.372 IDS|DDPLUS 762  SAUS46 
> KLOX 171520 /pMTRCQT
> Sep 17 16:33:55 pqutil:     6792 20020917152128.556 IDS|DDPLUS 772  SXUS86 
> KLOX 171520 /pOMRSGX
> Sep 17 16:34:04 pqutil:      144 20020917152134.808 IDS|DDPLUS 887  SRUS55 
> KVEF 171520 /pRR3VEF
> Sep 17 16:34:05 pqutil:      970 20020917152139.970 IDS|DDPLUS 923  WWUS82 
> KMLB 171521 /pSPSMLB
> Sep 17 16:34:05 pqutil:      492 20020917152140.520 IDS|DDPLUS 953  NTUS96 
> KNCF 171521 /pWTSNCF
> Sep 17 16:34:06 pqutil:      492 20020917152140.623 IDS|DDPLUS 958  NTUS99 
> KNCF 171521 /pTSTNCF
> Sep 17 16:34:06 pqutil:     4302 20020917152140.779 IDS|DDPLUS 968  NZUS11 
> KBYZ 171521 /pWRKRSA
> Sep 17 16:34:08 pqutil:     1344 20020917152146.076 IDS|DDPLUS 018  FXUS63 
> KGRR 171521 CCA /pAFDGRR
> Sep 17 16:34:10 pqutil:    23020 20020917152149.758 IDS|DDPLUS 043  FPUS55 
> KBYZ 171521 AAA /pZFPBYZ
> Sep 17 16:34:11 pqutil:     1325 20020917152156.601 IDS|DDPLUS 104  FGUS52 
> KCHS 171521 /pRVDCHS
> Sep 17 16:34:12 pqutil:     2445 20020917152156.027 IDS|DDPLUS 112  SRUS81 
> KPHI 171521 /pRRAPHL
> Sep 17 16:34:12 pqutil:      108 20020917152157.714 IDS|DDPLUS 140  SRUS56 
> KOTX 171521 /pRR3OTX
> Sep 17 16:34:13 pqutil:       85 20020917152202.447 IDS|DDPLUS 164  SRUS54 
> KSHV 171521 /pRR3SHV
> Sep 17 16:34:20 pqutil:      605 20020917152203.243 IDS|DDPLUS 188  FNUS72 
> KTAE 171521 /pSMFTLH
> Sep 17 16:34:21 pqutil:      109 20020917152203.567 IDS|DDPLUS 194  SRUS55 
> KLKN 171521 /pRR3LKN
> Sep 17 16:34:33 pqutil:     2453 20020917152211.970 IDS|DDPLUS 287  FZUS63 
> KLOT 171521 CCA /pGLFLM
> Sep 17 16:34:33 pqutil:      591 20020917152211.997 IDS|DDPLUS 292  NZUS50 
> KBYZ 171521 /pWRKCCF
> Sep 17 16:34:34 pqutil:      449 20020917152213.421 IDS|DDPLUS 370  FPUS74 
> KLZK 171521 /pNOWLIT
> Sep 17 16:34:34 pqutil:     2918 20020917152213.448 IDS|DDPLUS 372  FOUS61 
> KTAR 171521 /pFFGME
> Sep 17 16:34:34 pqutil:      235 20020917152213.653 IDS|DDPLUS 388  FTUS44 
> KMEG 171521 AAB /pTAFMEM
> Sep 17 16:34:41 pqutil:     3007 20020917152218.252 IDS|DDPLUS 509  FOUS61 
> KTAR 171521 /pFFGMA
> Sep 17 16:34:41 pqutil:     1782 20020917152218.324 IDS|DDPLUS 524  FXUS64 
> KBRO 171521 /pAFDBRO
> Sep 17 16:34:41 pqutil:     3695 20020917152223.213 IDS|DDPLUS 606  FOUS71 
> KTAR 171521 /pFFHHFD
> Sep 17 16:34:41 pqutil:     5265 20020917152223.234 IDS|DDPLUS 610  FOUS61 
> KTAR 171521 /pFFGNY
> Sep 17 16:34:41 pqutil:      125 20020917152224.379 IDS|DDPLUS 700  SRUS55 
> KBOI 171521 /pRR3BOI
> Sep 17 16:34:42 pqutil:     3146 20020917152229.065 IDS|DDPLUS 769  FZPN03 
> KNHC 171522 /pHSFEP2
> Sep 17 16:34:43 pqutil:      492 20020917152234.517 IDS|DDPLUS 917  NTUS96 
> KNCF 171522 /pWTSNCF
> Sep 17 16:34:44 pqutil:     1297 20020917152235.979 IDS|DDPLUS 032  SRUS44 
> KBRO 171522 /pRVABRO
> Sep 17 16:34:44 pqutil:       99 20020917152239.566 IDS|DDPLUS 067  SRUS52 
> KFFC 171522 /pRR3ATL
> Sep 17 16:34:44 pqutil:      492 20020917152239.570 IDS|DDPLUS 068  NTUS99 
> KNCF 171522 /pTSTNCF
> Sep 17 16:34:44 pqutil:      289 20020917152240.176 IDS|DDPLUS 116  SRUS54 
> KTUA 171522 /pRR2OKC
> ------------
> 
> No joy in mudville yet....later,
> 
> -Mike
> 
> 
> 
> 
> 
> 
> 
> 
> At 09:55 AM 9/17/2002 -0600, Jeff Weber wrote:
> >Hi Mike, 
> >
> >aeolus is up currently...
> >
> >aeolus.ucsd.edu> ldmadmin watch
> >(Type ^D or ^C when finished)
> >Sep 17 15:50:27 pqutil:     2706 20020917150418.944 IDS|DDPLUS 125  WTNT45
> >KNHC 171504 /pTCDAT5
> >Sep 17 15:50:27 pqutil:     1653 20020917150419.024     HDS 131  SDUS85
> >KABQ 171500 /pDPAFDX
> >Sep 17 15:50:27 pqutil:      149 20020917150419.062     HDS 134  NXUS64
> >KLCH 171503 /pGSMLCH
> >Sep 17 15:50:27 pqutil:      149 20020917150419.081     HDS 136  NXUS62
> >TJSJ 171502 /pGSMJUA
> >Sep 17 15:50:27 pqutil:    35928 20020917150419.178     HDS 140  ZHRK99
> >KWBE 171200 /mETA_84
> >Sep 17 15:50:27 pqutil:    35928 20020917150423.371     HDS 143  ZHRK93
> >KWBE 171200 /mETA_84
> >Sep 17 15:50:27 pqutil:       42 20020917150423.376 IDS|DDPLUS 144  SIVE25
> >RJTD 170900
> >Sep 17 15:50:27 pqutil:       42 20020917150423.378 IDS|DDPLUS 145  SIVX25
> >RJTD 170900
> >Sep 17 15:50:27 pqutil:     1013 20020917150423.380 IDS|DDPLUS 146  UMZZ42
> >KAWN 171200 RRX
> >Sep 17 15:50:27 pqutil:      287 20020917150423.383 IDS|DDPLUS 147  FTKO51
> >RJTD 171400 CCA
> >Sep 17 15:50:27 pqutil:      177 20020917150423.386 IDS|DDPLUS 149  UDEU02
> >EGRR 171450
> >Sep 17 15:50:28 pqutil:    35928 20020917150423.436     HDS 151  ZHRK95
> >KWBE 171200 /mETA_84
> >Sep 17 15:50:28 pqutil:    35928 20020917150423.516     HDS 154  ZHRK92
> >KWBE 171200 /mETA_84
> >Sep 17 15:50:28 pqutil:    35928 20020917150423.563     HDS 157  ZHRK91
> >KWBE 171200 /mETA_84
> >Sep 17 15:50:29 pqutil:   172188 20020917150423.023    FSL2 000
> >FSL.NetCDF.NOAAnet.windprofiler.06min.20022601454.*
> >Sep 17 15:50:29 pqutil:    35928 20020917150423.599     HDS 158  ZHRK90
> >KWBE 171200 /mETA_84
> >Sep 17 15:50:29 pqutil:      119 20020917150423.681 IDS|DDPLUS 165  SXXX40
> >KWAL 171449
> >Sep 17 15:50:29 pqutil:      132 20020917150423.692 IDS|DDPLUS 166  SXNE55
> >KWAL 171449
> >Sep 17 15:50:29 pqutil:      167 20020917150423.695     HDS 170  SACN94
> >CWAO 171400 RRD
> >Sep 17 15:50:29 pqutil:      113 20020917150423.696 IDS|DDPLUS 168  UDEU02
> >EGRR 171450
> >Sep 17 15:50:29 pqutil:      110 20020917150423.697 IDS|DDPLUS 169  SACN89
> >CWAO 171400 RRE
> >Sep 17 15:50:29 pqutil:      100 20020917150423.698 IDS|DDPLUS 171  SACN91
> >CWAO 171400 RRD
> >Sep 17 15:50:29 pqutil:       94 20020917150423.699 IDS|DDPLUS 172  SNTC01
> >LFPW 162300
> >Sep 17 15:50:29 pqutil:       94 20020917150423.699 IDS|DDPLUS 173  SNTC01
> >LFPW 162200
> >
> >
> >Tho they seem a bit behind ~46 minutes...
> >
> >Darn!
> >
> >Sunny89 is not working properly as a failover..?
> >
> >Keep me posted!
> >
> >
> >-Jeff
> >____________________________                  _____________________
> >Jeff Weber                                    address@hidden
> >Unidata Support                               PH:303-497-8676 
> >COMET Case Study Library                      FX:303-497-8690
> >University Corp for Atmospheric Research      3300 Mitchell Ln
> >http://www.unidata.ucar.edu/staff/jweber      Boulder,Co 80307-3000
> >________________________________________      ______________________
> >
> >On Tue, 17 Sep 2002, Mike Voss wrote:
> >
> >> Mornin' Jeff,
> >> 
> >> No data since about 5Z from aeolus.ucsd.edu. Can you see from their end if 
> >> they are down or not? I don't see anything with NOTIFYME. I guess I better 
> >> get my ldmfail script working correctly. I've failed over to Sunny89, but 
> >> I'm not getting anything....I imagine it has to figure out which products 
> >> to send...hmm, I don't recall the LDM having this problem before I changed 
> >> to 5.2.
> >> 
> >> -Mike
> >> 
> >> At 12:34 PM 9/16/2002 -0600, you wrote:
> >> >Hi Mike, 
> >> >
> >> >Thanks for adding rtstats, was going to do it while I was on, but thought
> >> >I would let you designate when rossby went down...
> >> >
> >> >No, it is not normal for these "other" feeds to lag..It did this when I
> >> >added motherlode as the HDS feed, odd...
> >> >
> >> >I suspect the IDS and NNEXRAD will catch up, but it is curious why they
> >> >take so long to fire up, this may be part of the HDS problem from aeolus.
> >> >
> >> >Still investigating..
> >> >
> >> >Thanks for adding the rtstats, this will help all of us better diagnose
> >> >events at SJSU!
> >> > 
> >> >More later,
> >> >
> >> >-Jeff
> >> >____________________________                  _____________________
> >> >Jeff Weber                                    address@hidden
> >> >Unidata Support                               PH:303-497-8676 
> >> >COMET Case Study Library                      FX:303-497-8690
> >> >University Corp for Atmospheric Research      3300 Mitchell Ln
> >> >http://www.unidata.ucar.edu/staff/jweber      Boulder,Co 80307-3000
> >> >________________________________________      ______________________
> >> >
> >> >On Mon, 16 Sep 2002, Mike Voss wrote:
> >> >
> >> >> Jeff,
> >> >> 
> >> >> I added " rtstat" to ldmd.conf.
> >> >> 
> >> >> When I restarted the LDM, HDS started flowing right away, but
> >> >> IDS|NEXRAD..etc is not flowing. It's like the program is having
> >> >> trouble syncing up product queue info to see what products should be
> >> >> sent. Is this normal? In my experience it is not normal.
> >> >> 
> >> >> Data did flow well over the weekend even though our network still has
> >> >> a bottle neck with large packets at 130.65.80.1. This problem is still
> >> >> being looked at by network engineers...no answer yet on why it's
> >> >> happening or what the fix is. I think the data flow has been able to
> >> >> overcome this because the rest of the network between here and
> >> >> UCSD/UCAR has been fine.
> >> >> 
> >> >> Gotta go hack away and get my data flowing. Oh, I see IDS|NEXRAD data
> >> >> is flowing again, but with a 30 minute latency. Should that really
> >> >> happen when I just stop and start the LDM? It was flowing with no
> >> >> latency previous to the restart.
> >> >> 
> >> >> Thanks,
> >> >> 
> >> >> -Mike
> >> >> 
> >> >> 
> >> >> 
> >> >> At 10:12 AM 9/16/2002 -0600, Jeff Weber wrote:
> >> >> >HI Mike, 
> >> >> >
> >> >> >Data seems to be more timely, but we are coming off the weekend..
> >> >> >
> >> >> >Will you please add:
> >> >> >
> >> >> >exec    "rtstats -h rtstats.unidata.ucar.edu"
> >> >> >
> >> >> >in your ldmd.conf file.
> >> >> >
> >> >> >And then start and stop..Do this at your convenience.
> >> >> >
> >> >> >
> >> >> >You can place this directly under:
> >> >> >
> >> >> >exec    "pqbinstats"
> >> >> >
> >> >> >This will allow us to more closely monitor traffic, and you can access
> >> >> >these stats as well at:
> >> >> >
> >> >> >http://www.unidata.ucar.edu/staff/chiz/latency/rtstats/index2.shtml
> >> >> >
> >> >> >Thanks,
> >> >> >
> >> >> >-Jeff
> >> >> >____________________________                  _____________________
> >> >> >Jeff Weber                                    address@hidden
> >> >> >Unidata Support                               PH:303-497-8676 
> >> >> >COMET Case Study Library                      FX:303-497-8690
> >> >> >University Corp for Atmospheric Research      3300 Mitchell Ln
> >> >> >http://www.unidata.ucar.edu/staff/jweber      Boulder,Co 80307-3000
> >> >> >________________________________________      ______________________
> >> >> 
> >> >> --------------------------
> >> >> Mike Voss                                
> >> >> Department of Meteorology               
> >> >> San Jose State University                        
> >> >> One Washington Square                      
> >> >> San Jose, CA 95192-0104   
> >> >>              
> >> >> 408.924.5204 voice
> >> >> 408.924.5191 fax   
> >> >> 
> >> >> 
> >> 
> >> --------------------------
> >> Mike Voss                                
> >> Department of Meteorology               
> >> San Jose State University                        
> >> One Washington Square                      
> >> San Jose, CA 95192-0104   
> >>              
> >> 408.924.5204 voice
> >> 408.924.5191 fax   
> >> 
> >> 
> 
> --------------------------
> Mike Voss                                
> Department of Meteorology               
> San Jose State University                        
> One Washington Square                      
> San Jose, CA 95192-0104   
>              
> 408.924.5204 voice
> 408.924.5191 fax   
> 
> 


NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.