North American Network Operators Group
Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical

Re: UUNET connectivity in Minneapolis, MN

  • From: Warren Kumari
  • Date: Fri Aug 12 13:01:40 2005

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

So I am standing in a datacenter fiddling with some fiber and listening to an electrician explaining to the datacenter owner how he has just finished auditing all of the backup power systems and that the transfer switch will work this time (unlike the last 3 times). This is making me a little nervous, but I keep quiet (unusual for me)... Electrician starts walking out of the DC, looks at the (glowing) Big Red Button (marked "Emergency Power Off") and says "Hey, why ya'll running on emergency power?" and presses BRB. Lights go dark, disks spin down, Warren takes his business elsewhere!

This is the same DC that had large basement mounted generators in a windowless building in NYC. Weeks before the above incident they had tried to test the generator (one of the failed transfer switch incidents), but apparently no one knew that there were manual flues at the top of the exhausts.... Carbon monoxide, building evacuated...

Warren

On Aug 12, 2005, at 8:27 AM, [email protected] wrote:

On Fri, 12 Aug 2005 06:50:47 CDT, "James D. Butt" said:
Unless there is some sort of crazy story related to why a service provider
could not keep the lights on, this should have not been an issue with
proper operations and engineering.

So a while ago, we're in the middle of some major construction to put in
infrastructure for a supercomputer. Meanwhile, as an unrelated project we
installed a new diesel backup generator to replace an older generator that was
undersized for our current systems, and take several hours of downtime
on a Saturday to wire the beast in.

The next Friday, some contractors are moving the entrance to our machine room
about 30 feet to the right, so you don't walk into the middle of the
supercomputer. Worker A starts moving a small red switch unit from its
location next to where the door used to be to its new location next to where
the door was going to be. Unfortunately, he did it before double- checking with
Worker B that the small red switch was disarmed...

Ka-blammo, a Halon dump... and of course that's interlocked with the power,
so once the Halon stopped hissing, it was *very* quiet in there.....

Moral: It only takes one guy with a screwdriver.....

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (Darwin)

iD8DBQFC/NVFHSkNr4ucEScRAkc9AKCnwraT9DztjAConsyuBZ7wDs/bJACgyrWR
e2zcwlIffPxhTKfFJWm3T3A=
=qDyJ
-----END PGP SIGNATURE-----