|
North American Network Operators Group Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical Re: BBN (GTE) Suffers another major power problem.
> >If you're in a 24x7, customer-driven market, you should have either: > > > >a) enough go-juice (via fuel contractor) to go until the grid is on-line > > Did you read the previous email from jhawk? There is an > extremely large generator at one of the affected facilities > (read: cogeneration plant) but no amount of fuel helps if > the plant is damaged in the explosion. Unfortunately, the previous mail from jhawk went to BBN's local NANOG exploder and not to the public NANOG list :-( As long as I'm taking a point to clarify publically, I'll try to be complete. > p.s. No, we've have not deployed redundant power plants (yet) BBN has a number of facilities, some of which are colocated in telco facilities, and some of which are not. Those that are in telco facilities have (as you would, of course, expect) hours-to-days of backup power in the event of a catastrophic failure. Those that are not, however, do not. BBN has major facilities colocated with both MIT and Stanford, both of which have co-generation power, which is expected to be available should utility power failure. Of course, it is well-recognized, and recent history serves to enforce that quite well, that this doesn't help for many kinds of failures, especially if the distance between the backup power source and your equipment is large, with multiple possible points of failure. Sean Doran is perfectly correct when he points out that market economics are central here. When engineering redundancy, there's going to always be some number of customers that you are willing to let suffer in the event of a catastrophy. In some cases, maybe that number is zero. In other cases, perhaps it's some number of percentage points. It clearly doesn't make sense to engineer for the case where a disaster takes out hundreds of customers, leaving only one remaining :-) I'm not prepared to address what our outlook on this is, of course, however we generally feel that facilities with co-generation are less likely to be in horrible power positions than other facilities. Maybe not "less by a lot", but certainly "less". Unlike the Stanford case, however, while we did not have on-site generator power, we were certainly more prepared for it, and it in fact arrived, largely coincident with the return of utility power. That's the way it goes. Attached are some exerpts of summaries from the MIT which you might care for if you want nitty-gritty details. Folks on nanog who commented that having an on-site generator for outages being the way to go are certainly correct -- if your marketting economics justify them. I can't really speak to ours or our future plans, however. To answer some other questions, BBN does indeed have multiple POPs in the Cambridge/Boston area, and this only affected one of them. The suggestino that it was "physically impractical" to power the equipment in question (Chris Masto) is simply incorrect. Matt Ringel's comment that the power was out from 1730 through 2245 (EDT) is not completely correct (see below), as there was quite a long period of restored power in the middle, and the second outage was more widespread than the first. Note, also, that UPS power caused our equipment to run for quite a while longer than the numbers below indicate, but I don't have precise figures on that handy. The assertion that we had emergency generator backup at this location (Eric Osborne) was incorrect; a generator was brought in as the outage drew to a close, and co-generation was normally expected to be available. The initial mail (Patrick Chicas) implied that power to the building housing our POP was lost -- in reality the problem was roughly half of the city of Cambridge, MA. I think that just about covers all the bases. --jhawk Subject: Power Outage August 7 , 1997 Date: Fri, 8 Aug 1997 07:39:43 -0400 At approximately 5:35pm on August 7, 1997 a manhole explosion in the Kendall Sq area caused Cambridge Electric to loose Kendall Station. MIT lost all power and the gas turbine tripped. Power was fully restored at 7pm. At approximately 7:05pm, a second manhole explosion,caused 1 fatality, and injurys to 4 other Cambridge Electric utilitymen including a Cambridge Policeman. Putnam Station was also tripped and MIT lost all power again. At approximately 10:30pm, MIT had power restored to all buildings within our distribution system. Several East campus ( E28, E32, E42, E56, E60, NE43)and North/Northwest buildings( N42, N51/52, N57, NW10, NW12, NW14, NW15, NW17, NW20, NW21, NW22, NW30, NW61, NW62, W11, WW15) which are fed directly from Cambridge Electric, were restored by Cambridge Electric personal. Cambridge Electric is still sorting out the chain of events. At last discussions with them, a total of 3 manhole explosions had taken place. Additional information will be posted when available. [...] Subject: Power Problem Update Date: Fri, 8 Aug 1997 09:31:05 -0400 Friday morning, @ 0930 Power problem update: The MIT cogeneration facility was damaged during the power problems that occurred last night. As a result, the campus is operating totally on power supplied by Cambridge Electric. At this time, Cambridge Electric is stable, however, due to the events that occurred yesterday and early this morning, it is uncertain how reliable their distribution system will be until repairs are accomplished. MIT is working as fast as possible to get repair parts for the cogenerator and get it on line. It is expected that the parts will arrive this afternoon for immediate installation. [...]
|