Best practice in networking calls for two separate power sources. Rarely
will you see a short circuit cause simultaneous outages unless it manages
to kill two separate battery plants. OR, if they are powering from AC, they
are a mission critical asset, so they should have two power feeds, coming
from two different UPS's.
If they lacked diversity in sources, that's a problem and could lead to an
outage if they had a single circuit failure.
Or, they could have done router software or config updates on both routers
simultaneously, while they were swapped out.
The possibilities are finite, really, and I suspect they'll have someone
investigate.
gerry
On Tue, Feb 14, 2017 at 4:36 PM, <admin@xxxxxxxx> wrote:
>
> Ive been there too Kevin, but most interconnects from circuits to routers
> now use fiber optic jumpers in datacenters. For that exact reason. But we
> weren't there, could have been anything.
>
> Ray
>
> On Tuesday, February 14, 2017 5:29pm, "Kevin W. Thomas" <
> kwthomas@xxxxxxxxxxxxxxxxxx> said:
>
> > I'm far from a hardware expert, but I do know electrical shorts can go
> thru
> > cables and take out lots of hardware.
> >
> > Many years ago, I administered a PDP 11/44. The computer room took a
> hit from
> > lightning that fried a lot of computer boards. The surge went down all
> the
> > cable lines and fried a chip an almost all of the VT100's that were
> attached
> > to it. Distance to the computer room didn't matter.
> >
> > This sounds similar.
> >
> > Kevin W. Thomas
> > Center for Analysis and Prediction of Storms
> > University of Oklahoma
> > Norman, Oklahoma
> > Email: kwthomas@xxxxxx
> >
> >>Stonie is right on here. Another thing I find suspicious is that "Two
> routers
> >> failed simultaneously". The odds of that are next to impossible. More
> likely the
> >> routers responded improperly to an event.
> >>
> >>Ray Weber
> >>MA Skywarn
> >>
> >>On Tuesday, February 14, 2017 4:37pm, "Stonie R. Cooper" <
> stonewall@xxxxxxxxxxxxx>
> >> said:
> >>
> >>> I generally keep my comments off of this list, but the official
> >>> statement is not accurate. As you know, Gilbert, WRIP (NOAA Weather
> >>> Radio) was only partially up and that was WFO dependent - if an WFO
> >>> fired up their CRS, they could manually record hourly forecasts and
> >>> observations.
> >>>
> >>> Likewise, checking the tombstones via local forecast on weather.gov
> >>> resulted in "N/A" for most zip codes I tried during the outage.
> >>>
> >>> From an IT side, I would say this particular event should help
> >>> illustrate the foolhardy paradigm of "putting all your eggs in one
> >>> basket." HPC, NCEP, EMC . . . all were hit with this outage. NWWS,
> >>> EMWIN, and SBN were all down. A primary and secondary router paradigm
> >>> in the same physical location may save lots of money, but is an
> >>> incredible liability for data distribution.
> >>>
> >>> I have always advocated that a live secondary should be placed
> >>> geographically distant from the primary uplink and NCF . . . and KC has
> >>> legacy and national networking infrastructure already in place to make
> >>> it a likely candidate. Or Boulder, at ESRL.
> >>>
> >>> Not that my comments will merit any action, but if pointing out what
> >>> seems obvious to me gains traction, that would be great. I would even
> >>> start filling out my TPS cover sheets for all my memos, and coming in
> on
> >>> over the weekend to work. ("Office Space" reference)
> >>>
> >>> Stonie
> >>>
> >>> On 02/14/2017 03:17 PM, Gilbert Sebenste wrote:
> >>>> Here it is:
> >>>>
> >>>>
> >>>>
> >>>> NWS Statement on Cause of Outage on Feb. 13
> >>>> Feb. 14, 2017
> >>>>
> >>>> The National Weather Service experienced a failure of its AWIPS
> Network
> >>>> Control Facility communications network at 2:08 p.m. EST Feb 13. The
> >>>> outage, lasting two hours, 36 minutes, prevented us from fully
> >>>> distributing forecasts and warnings. During the outage, the public was
> >>>> able to access forecasts, watches and warnings by NOAA Weather Radio
> and
> >>>> the social media accounts of their local forecast office.
> >>>>
> >>>> Technicians quickly determined the cause of the problem was the
> >>>> simultaneous failure of two core communications routers - primary and
> >>>> backup - for the control facility due to a power problem. The routers
> >>>> were replaced and the system was restored to full service. We are
> still
> >>>> investigating what caused the power outage.
> >>>>
> >>>> The AWIPS communications system is a very reliable configuration and
> >>>> this is the first time both routers failed simultaneously.
> >>>>
> >>>> We are implementing additional communication pathways to the backup
> >>>> Network Control Facility to ensure that problems encountered in
> >>>> switching operations to this backup facility will not recur.
> >>>>
> >>>> ---
> >>>>
> >>>>
> >>>>
> >>>> *Gilbert Sebenste*
> >>>>
> >>>> Staff Meteorologist
> >>>>
> >>>>
> >>>>
> >>>> Environmental Health and Safety
> >>>>
> >>>> Labs for Wellness 154 | DeKalb, Illinois 60115
> >>>>
> >>>> 815-753-5492
> >>>>
> >>>> _gilbert@xxxxxxx <mailto:gilbert@xxxxxxx>_
> >>>>
> >>>> http://weather.admin.niu.edu <http://weather.admin.niu.edu/>
> >>>>
> >>>> Everyone. Home. Safely.
> >>>>
> >>>>
> >>>>
> >>>> NIU
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> _______________________________________________
> >>>> NOTE: All exchanges posted to Unidata maintained email lists are
> >>>> recorded in the Unidata inquiry tracking system and made publicly
> >>>> available through the web. Users who post to any of the lists we
> >>>> maintain are reminded to remove any personal information that they
> >>>> do not want to be made public.
> >>>>
> >>>>
> >>>> ldm-users mailing list
> >>>> ldm-users@xxxxxxxxxxxxxxxx
> >>>> For list information or to unsubscribe, visit:
> >>>> http://www.unidata.ucar.edu/mailing_lists/
> >>>>
> >>>
> >>> _______________________________________________
> >>> NOTE: All exchanges posted to Unidata maintained email lists are
> >>> recorded in the Unidata inquiry tracking system and made publicly
> >>> available through the web. Users who post to any of the lists we
> >>> maintain are reminded to remove any personal information that they
> >>> do not want to be made public.
> >>>
> >>>
> >>> ldm-users mailing list
> >>> ldm-users@xxxxxxxxxxxxxxxx
> >>> For list information or to unsubscribe, visit:
> >>> http://www.unidata.ucar.edu/mailing_lists/
> >>>
> >>
> >>
> >>_______________________________________________
> >>NOTE: All exchanges posted to Unidata maintained email lists are
> >>recorded in the Unidata inquiry tracking system and made publicly
> >>available through the web. Users who post to any of the lists we
> >>maintain are reminded to remove any personal information that they
> >>do not want to be made public.
> >>
> >>
> >>ldm-users mailing list
> >>ldm-users@xxxxxxxxxxxxxxxx
> >>For list information or to unsubscribe, visit:
> >> http://www.unidata.ucar.edu/mailing_lists/
> >>
> >
>
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web. Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> ldm-users mailing list
> ldm-users@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
>
--
Gerry Creager
NSSL/CIMMS
405.325.6371
++++++++++++++++++++++
“Big whorls have little whorls,
That feed on their velocity;
And little whorls have lesser whorls,
And so on to viscosity.”
Lewis Fry Richardson (1881-1953)
