North American Network Operators Group

Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical

Re: BBN outage

  • From: Patrick J. Chicas
  • Date: Mon Oct 14 21:50:57 1996

On Sat, 12 Oct 1996, Nathan Stratton wrote:

> On Fri, 11 Oct 1996, Rob Liebschutz wrote:
> 
> > I just spoke to their NOC and was told that a power switch that is
> > supposed to be able to switch them between 3 different power utilities
> > failed at 12:30 this morning (friday).  They bought and installed 2
> > large diesel generators today and are hoping to be back up using the
> > generators within 30 minutes.

Greetings,

I don't understand how they could have 3 different power systems fail
without a serious operations procedural error.

No one buys more than one generator and it appears that they didn't
have one at all so, that's ruled out. They have one utility company,
and let's say two UPS's to feed the dual power bussed routers. At best
they have a DC plant.

Overloaded UPS's are easy to spot. Weak batteries are found during
routine, in-service tests.

So.. It looks like they abhorently ignored thier power situation. This is
an "act of stupidity", not an "act of god" as one other person mentioned.
 
> Yep this happens all the time, the transfer switch dies and then you are
> screwed because you don't switch to backup power.

In 18 years of telecom management I have never seen a "Transfer Switch" as
a component failure. I have seen quite a few overloaded battery strings,
and UPS's backed by rusty generators.

> Your UPS system then run
> out of power and you are dead. That is why we are building a manual
> maintenance wraparound around the UPS AND the transfer switch so that if
> they switch does die you can manually have some guy bypass the switch.

All quality Transfer Switches should have manual activation as a root
function. Even a relatively small 15kw transfer switches automatic
functions work to move a manual switch.

As a rule, power system maintenance for critical equipment should comprise
the following:

UPS- Never exceed 80% load. Replace batteries (good or not) per
manufactures guidlines. Initiate load transfer tests once per quarter.

Batteries- Preform cell maintenance quarterly, to include individual cell
voltage and gravity tests and, the surface cleaning of all terminal
hardware.

Generators- Find a good generator maintenance contractor for routine
maintenace needs. Exercise the unit under load each month.

Regards

 Patrick J. Chicas
 Email: [email protected]
 URL: http://www.Off-Road.com 
 --------------------------------
 The Off-Road Center of The 'Net!
 


- - - - - - - - - - - - - - - - -