NEXRAD Level II Maintenance Notification - Wednesday October 12th, 2016 (ROC)

ROC engineering will be performing routine maintenance activities to the ROC portion of the Level II Network.  The activity is a software & security upgrade required on the gateway for the ROC Level II cluster.
 
 Start Time: 10/12/2016 1:00pm (Central Time)
 Duration: 1 Hour
 Event: ROC Software & Security Updates (NL2 software release 7.1)
 
 The activity will take about 1 hour to complete and will effect Data flow of the "Alternate Level II Path" for up to 10 minutes.  (RDS Path)
 
 NOTE:  (This Morning's outage.)

The ROC servers were set as AGGREGATE and the servers in Silver Spring were locked to allow the configuration beyond a one hour default roll back.  A fire drill within the ROC facility this morning also tested the alarm system which cuts power to key areas, protecting 1st responders and to prevent additional damage to equipment.  The "worst case scenario" then ensued that shut down power to the computer room long enough to let the NL2 UPS backup expire while the Silver Spring servers were commanded not to take over.  We are working our internal processes to recognize and prevent this from happening again.

While we were coming back up, I rolled the aggregation function back to the TOC servers to support the software load activity.

THIS MAY ALSO CAUSE a "known" latency issue to resurface with the RDS feed.  We've been working with NCO on this issue (since Aug 5th) that surfaces when TOC is AGGREGATE (the system default) and NCO has performed network adjustments that may mitigate or resolve this issue.  If the 30 minute RDS latency issue returns and/or becomes problematic operationally for any client, the ROC can roll back to the Norman cluster. 

I appreciate your patience as we continue to resolve the latency issue. 

For those of you tracking this issue it's OneNWSnet issue: ONCM20160919004720

 All system upgrades are subject to NCEP Critical Weather Day notification.
 
 http://www.nco.ncep.noaa.gov/pmb/cwd/
 
 If there is any emergency noted during this event, I can be reached at (405) 820-3646

 If there is an issue with the latency, my (8-5) desk number is (405) 573-3418

And as always the ROC HOTLINE will assist as required or just track me down... (800)643-3363

 
 Thanks...
 
    ...Don

-- 
**********************************************************************
*Donald J. Horvat               *  ____                              *
*                               * /    \     _____     _______  _____*
*Sr. Systems Engineer           *(      )   / ___  )  / ___  / / ___/*
* Support Contractor            * \____/   / /__/ /  / /  / / / /    *
*WSR-88D Radar Operations Center*  |\/|   / ___  <  / /  / / / /     *
*                               *  |/\|  / /   / / / /__/ / / /___   *
*1313 Halley Circle             *  |\/| /_/   /_/ /______/ /_____/   *
*Norman, OK 73069               *  |/\|                              *
*                               *   WSR-88D RADAR OPERATIONS CENTER  *
**********************************************************************
*Phone: (405) 573-3418          * e-mail: address@hidden   *
*FAX:   (405) 573-3340          *                                    *
**********************************************************************