North American Network Operators Group

Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical

Re: San Francisco Power Outage

  • From: Jeff Aitken
  • Date: Wed Jul 25 10:51:06 2007

On Tue, Jul 24, 2007 at 09:57:09PM -0500, Brandon Galbraith wrote:
> It appears that 365 is using the Hytec Continuous Power System [
>], which is a motor,
> generator, flywheel, clutch, and Diesel engine all on the same shaft. They
> don't use batteries.

Yes.  I used to work for the company that originally built the 365 Main
datacenter and remember touring it near the end of the construction phase.
The collection of power units up on the roof was impressive, as were the
seismic isolators in the basement.

But even when you try and do everythying right Murphy usually finds a way
to sneak up behind you and whisper "BOHICA" in your ear.  For example, we
had a failure at another datacenter that uses Piller units, which operate
on the same basic principle as the Hitec ones.  While running on generator
one of the engines overheated due to an oil-flow problem and threw a rod.
When the on-duty electrician responded to the alarm, there were red-hot
chunks of engine *outside* of the enclosure, and there was a hole in the
side of the unit large enough to stick your arm in.  The facility manager
kept the damaged piston as a momento. :-)

I don't remember whether this was due to a design flaw, improper
installation, or what, but the important points are that (1) this is the
real world and shit happens, and (2) it wasn't until the generator was
worked long enough that the reduction in oil flow caused enough friction
to trigger a catastrophic failure.  I.e., there's no guarantee that you
will catch this kind of problem in your monthly tests.

On Tue, Jul 24, 2007 at 05:39:34PM -0700, George William Herbert wrote:
> Unfortunate real-world lesson: there is a functional difference between
> pushing the UPS test cutover button, and some of the stuff that can happen
> out on the power lines (including rapid voltage swings, harmonics, etc).