North American Network Operators Group

Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical

Re: WSJ: Big tech firms seeking power

  • From: Alex Rubenstein
  • Date: Fri Jun 16 17:25:42 2006

On Fri, 16 Jun 2006, Matthew Crocker wrote:

I wonder just how much power it takes to cool 450,000 servers.
450,000 servers * 100 Watts/Server = 45,000,000 watts / 3.413 watts/BTU = 13.1 Million BTU / 12000 BTU/Ton = 1100 Tons of cooling
Error: you MULTIPLY 3.413 to go from watts to BTU, not divide. It's be more like 154,000,000 BTU, /12000 or 12,798 tons.

Also at 100 watts, you are assuming Celerons with single hard drives. We see more like 120 to 240 depending on config. 100 would be low.

A 30 Ton Liebert system runs about 80 amps @ 480 volts or 38400 watts, you'll need at least 40 or them to cool 1100 tons which is 1536 Kw * 24 hours * 7 days * 4.3 weeks = 1,110,000 KwH/month * $0.10/KwH = $111,000 /month in cooling.
80 amps @ 480 is 80 * 480 * 1.73, or 66 kw. However, they don't draw that much. A 30 ton unit, worst case (115 degrees outside across the condensor) will be about 50 kw, assuming you do not have humidification or reheats turned on.

Second issue: you are assuming 100% cooling efficiency, or, in other words, that you'd have perfect airflow, perfect air return, etc. Never happens, especially when you have customers who are idiots.

Third issue: you are assuming there is no heat loss or gain in the structure of the building. This could be very significant. Let's assume it's not.

It's likely in an environment like this, you'd have more like 14000 tons. 14000 / 30 = 466 units, @ 50 kw/unit, 23,300,000 watts, / 1000 * 24 * 30.4375 (avg days in a month) = 17,020,000 kw-hrs, @ $0.12 (more likely with todays fuel prices unless you are in Kentucky) $2,042,400/month.

Also, don't forget the original 450,000 servers at 100 watts (45 mw) would be $3,944,700/month in power. Also, 450,000 1U servers at 40/rack would be 11,250 racks, which at 10 sq-ft a rack would be 112,000 sq-ft of datacenter floor space (triple or, more likely, quadruple that for space for HVAC, generators, switchgear, UPSs, etc). That'd be 500,000 sq-ft at minimum.

Total is $5,987,000/mon, but you haven't ROIed the millions in electrical gear (think big: this is about 68 megawatts; $250k/each for a 2 mw generator (you'd need 40, $10 mm), $50k/each for a 500 kva UPS (you'd need 80 $4mm), millions in panels, breakers, piping, copper wire (700% increase in copper pricing in the last 24 months, people), etc. Oh, and 466 liebert 30 ton HVAC's, probably $25 to $40k/ea installed ($11 million). Oh, and no one has installed it yet, and you haven't paid rent on the facility that will take 2 years to build with probably 100's of workers saleries.

Take $6mm/month, divide by 450,000 servers, $13.33/month/server.

Oh, and 68 Megawatts over 112k ft of floor space is 607 watts/ft. Thats about 6 times what most centers built in the last couple years are built at.

But wait, there is more. Just a point of comparison -- Oyster Creek Nuclear Power generation plant, located here on the Jersey Shore, produces 636 megawatts. You'd take one-tenth of that capacity -- in a bulding that would sit on a 10 or 20 acre chunk of land. I put this into the 'unlikely' category. The substation alone to handle stepping 68 mwatts from transmission to 480v would be probably 4 acres. And, 68 megawatts of power at 480 volts 81,888 amps. A typicall 200,000 sq-ft multi-tenant office building has 1600 amps of service; this would be the equivalent of 50 buildings.

Having fun yet?

A 30 ton liebert takes about 30 sq-ft of floor space; 466 of them would be 13,980 sq-ft. If you use a drycooler system, they are about 100 sq-ft, and youd need 233 of them (60 ton DDNT940's), 23,300 sq-ft of roof space. Each of those weighs 2,640 pounds, for a total of 615,000 pounds, or 308 tons (of weight, not HVAC capacity). I won't even spend the CPU cycles figuring out how many gallons of glycol this would bem but probably a good guess would be about 50,000 gallons. That'd be about a quarter-million dollars in glycol.

I'm tired now, time to climb back in my hole. In other words, don't get me started on the datacenter density issue.

Alex Rubenstein, AR97, K2AHR, [email protected], latency, Al Reuben
Net Access Corporation, 800-NET-ME-36,