[nws-changes] 20070821: MODIS outage

[Linda Miller]

Subject:
ESPC Data Outage (MODIS data UPDATED): issued 08-21-07, 1055 UTC
From:
ESPCoperations <ESPCoperations at noaa.gov>
Date:
Tue, 21 Aug 2007 06:54:49 -0400
To:
_NESDIS OSDPD ESPC OUTAGE Notification <ESPC.Notification at noaa.gov>

Details of the Outage:

ESPC is not receiving the MODIS data. The production server for MODIS 
suffered a severe hardware failure
yesterday August 15, 2007.   We're down to 75% of processors and 50% for 
RAID throughput. At this level
we are unable to even keep up with the incoming data from NASA.  We are 
rerouting systems as fast as we can
to try and restore minimal functionality.  We are simultaneously 
attempting to get a repair order started on the hardware.
At this point there is no ETA for when any stable data production will 
be available.
                                                                         
(Update) 08-18-07) 1651 UTC
Received an e-mail from Dale Zbesheski stating, K. Spreitzer and P. 
Haggerty report that they were able to get the backup servers at Goddard 
up to process all level 0 data and critical areas. AIRS and AMSRE are 
down until the data is redirected by NASA to these servers.
                                                                          
(Update) 08-18-07) 2050 UTC
K. Spreitzer and P. Haggerty report,  some  of  the  MODIS  applications 
are working now.
                                                                            
(Update) 08-20-07) 1815 UTC
Data AEDOS cannot push data to either SOAP or Bubbles.  We've been 
working this morning in attempt to get data pushed to the NSOF servers. 
Affected by the Outage:
                                                                              
(Update) 08-21-07) 1055 UTC

The problem seems to be with the RAID drive itself.  We will have to
call the specialist tomorrow morning.EDOS was able to connect to our 
NSOF servers this afternoon.  Our data
connection appears to be handling the data to and from NSOF.
Currently, we are re-routing the data back to SOAP our backup server for
spot processing and distribution.  The areas that are processed will be
sent on to NANUK.


                                                                           
MODIS (Moderate Resolution Imaging Spectrometer)

Date and Time of the Outage:

8/15/07  1100 UTC

Length of the Outage:

Until further notice.

Contact Point for Questions:
ESPC Operations  OCL (Operations Crew Leader),  301-817-3880