North American Network Operators Group
Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical

Re: History of the EPO (Emergency Power Off)

  • From: Robert Boyle
  • Date: Wed Jul 25 23:23:55 2007

At 08:10 PM 7/25/2007, Sean Donelan wrote:
Sometimes you need to revisit the rules. For example, for folks
thought having automatic water sprinklers in data centers was a bad thing. Slowly folks have started to rethink it, and now automatic sprinklers are
found in more data centers. I don't have hard data, but my experience
is there have been fewer outages from accidental sprinkler discharges
than from accidental EPO activations.

There was an interesting study conducted by the US Air Force about fires and other failure modes in computing facilities protected with Halon/FM200/FE227 vs. dry pipe preaction. I know I saved the PDF, but I can't seem to find it at the moment. If my memory is correct, it boiled down to the fact that there had only been two fire incidents at all US Air Force installations and both were due to (surprise, surprise) human factors. One was a stray incendiary munition which breached the datacenter and other was due to a Jet A fuel spill and fire - which is odd because it is hard to ignite kero, diesel, jet A without atomization. The point of the study was that there was zero damage over a 30 year period from water based fire protection systems and I suspect it was pretty handy to have sprinklers when both datacenter fires happened. The munition breach of the physical structure would have rendered any gas based fire suppression system ineffective.

In theory, I'm not a big fan of EPOs due to the "Is this the button to exit/open the door?" problem. One of our redundant 150KVA UPS units caught fire a couple years ago, the input breaker became the EPO since the on-board front panel EPO was completely ineffective (and it still would have been ineffective had it been connected to an external EPO button.) That incident prompted a design change in all of our new datacenter power systems since and all existing systems were also updated. Now all UPS units have separate input and bypass breakers and feeds. Previously we used a single feed, but you can't isolate a burning UPS without dropping your attached load when they share a single breaker and are tied together inside the unit where the fire is happening. Having discrete A & B power systems is also a very good thing!

Many years ago when we were much, much smaller, the EPO was wired to a special EPO circuit breaker on the main panel which fed the subpanel for the datacenter room. A short on that breaker was like pressing the "test" switch on a GFCI breaker. Do most people who do have functional (as opposed to decorative) EPO buttons have them connected to the building/suite mains disconnect? or to the output of your UPS units? to a special EPO panel which trips the EPO cutoffs on other units?

-Robert


Tellurian Networks - Global Hosting Solutions Since 1995
http://www.tellurian.com | 888-TELLURIAN | 973-300-9211
"Well done is better than well said." - Benjamin Franklin