North American Network Operators Group

Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical

FINAL: MFS Outage: 6:30PM PDT 4/18/97 - ~1:30AM 4/19/97

  • From: Jay Adelson
  • Date: Sat Apr 19 04:39:56 1997

This final incident report is being sent at 1:14AM 4/19/97:

UPDATE:  (7:30PM PDT)

Starting at approximately 6:30PM PDT, MFS's power
unit on their fiber mux located at the Digital Internet
Exchange is resetting every 30 seconds to 1 minute.

All MFS fiber facilities are down during the power
reset.  This includes all MFS/Worldcom DS3 and T1
circuits entering the PAIX collocate facility.

The IX Operations staff (Digital Internet Exchange
Operations) believes that the problem is due to a
bad rectifier on the fiber mux's power rack.

We are currently waiting for MFS to provide a dispatch
to fix this problem.

UPDATE: (11:00PM PDT)

At approximately 7:30PM, MFS arrived on site.
We offered them clean power from our in-house
UPS and rectifiers at this time, which would have
prevented the rapid, regular resetting of their
line cards.  They were not interested in the power
at this time.

By 8:30PM, they were convinced their rectifier
was the cause of their power loss.  They still are
not interested in our power.

By 10:15, they had restored power to their rectifier.
Many of their line cards, including an OC48, did not
return back to an operational state, likely due to
the rapid resetting previously.  Again we offered our
power, however now they definitely were convinced it
was unnecessary as their rectifiers were functioning.
Instead, they are forcing us to wait until next week.

As of 11:00, the line cards are still not functioning.

UPDATE: 12:10AM 4/19/97

As of 11:45PM, the MFS technicians decided to go
ahead and begin cut over to our power so that at
least the power is stable while they work on the
line cards.  They are currently working on this cutover.

UPDATE: 1:14AM 4/19/97

By 12:15AM they changed their minds on the power 
cutover again.  The electricians installed two
redundant rectifiers and two additional batteries
to the system, which should provide power until
the cutover to the Digital Internet Exchange power
next week.  Also by this point they have determined
that the problem with the line cards is due to the
fact that they lost their configuration during the
power fluctuations.  They begin reprogramming them
at this time.

At 12:35AM, MFS customers begin reporting their
services returning to an operational state.

At 1:00AM, the MFS representative completes all
programming.  He says that the circuits will return
to service, one by one, using a "time lapse" system
over the next 30 to 35 minutes.  By 1:35AM, all
services should be restored.

At 1:25AM, MFS left the site saying all circuits 
that were "called in" by their customers were on-line.

If by 2:35AM your services are not restored, please
contact MFS and give them your circuit I.D.  We are
available for any remote hands requirements in the
facility at 1-800-209-9898.  
Thank you,

-Jay Adelson

-----------------------------------------------------------
Jay Adelson                  |    Email: [email protected]
Operations Manager           |    Voice: (415) 617-3607
Digital Internet Exchange    |      Fax: (415) 617-3233
-----------------------------------------------------------
               Secure. Reliable. Connected.

- - - - - - - - - - - - - - - - -