North American Network Operators Group Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical Re: FINAL: MFS Outage: 6:30PM PDT 4/18/97 - ~1:30AM 4/19/97
Well done. -- Rusty --------------------------------------------------------------------- | MCI Communications | | Internet MCI [email protected] | ---------------------| http://infopage.mci.net |--------------------- > This final incident report is being sent at 1:14AM 4/19/97: > > UPDATE: (7:30PM PDT) > > Starting at approximately 6:30PM PDT, MFS's power > unit on their fiber mux located at the Digital Internet > Exchange is resetting every 30 seconds to 1 minute. > > All MFS fiber facilities are down during the power > reset. This includes all MFS/Worldcom DS3 and T1 > circuits entering the PAIX collocate facility. > > The IX Operations staff (Digital Internet Exchange > Operations) believes that the problem is due to a > bad rectifier on the fiber mux's power rack. > > We are currently waiting for MFS to provide a dispatch > to fix this problem. > > UPDATE: (11:00PM PDT) > > At approximately 7:30PM, MFS arrived on site. > We offered them clean power from our in-house > UPS and rectifiers at this time, which would have > prevented the rapid, regular resetting of their > line cards. They were not interested in the power > at this time. > > By 8:30PM, they were convinced their rectifier > was the cause of their power loss. They still are > not interested in our power. > > By 10:15, they had restored power to their rectifier. > Many of their line cards, including an OC48, did not > return back to an operational state, likely due to > the rapid resetting previously. Again we offered our > power, however now they definitely were convinced it > was unnecessary as their rectifiers were functioning. > Instead, they are forcing us to wait until next week. > > As of 11:00, the line cards are still not functioning. > > UPDATE: 12:10AM 4/19/97 > > As of 11:45PM, the MFS technicians decided to go > ahead and begin cut over to our power so that at > least the power is stable while they work on the > line cards. They are currently working on this cutover. > > UPDATE: 1:14AM 4/19/97 > > By 12:15AM they changed their minds on the power > cutover again. The electricians installed two > redundant rectifiers and two additional batteries > to the system, which should provide power until > the cutover to the Digital Internet Exchange power > next week. Also by this point they have determined > that the problem with the line cards is due to the > fact that they lost their configuration during the > power fluctuations. They begin reprogramming them > at this time. > > At 12:35AM, MFS customers begin reporting their > services returning to an operational state. > > At 1:00AM, the MFS representative completes all > programming. He says that the circuits will return > to service, one by one, using a "time lapse" system > over the next 30 to 35 minutes. By 1:35AM, all > services should be restored. > > At 1:25AM, MFS left the site saying all circuits > that were "called in" by their customers were on-line. > > If by 2:35AM your services are not restored, please > contact MFS and give them your circuit I.D. We are > available for any remote hands requirements in the > facility at 1-800-209-9898. > Thank you, > > -Jay Adelson > > ----------------------------------------------------------- > Jay Adelson | Email: [email protected] > Operations Manager | Voice: (415) 617-3607 > Digital Internet Exchange | Fax: (415) 617-3233 > ----------------------------------------------------------- > Secure. Reliable. Connected. > - - - - - - - - - - - - - - - - -
|