North American Network Operators Group

Date Prev | Date Next | Date Index | Thread Index | Author Index | Historical

Re: 2006.06.06 NANOG-NOTES IDC power and cooling panel

  • From: Thomas Leavitt
  • Date: Thu Jun 08 14:35:52 2006

Great notes! Very educational, I'm filing them all away.

Too bad APC wasn't a part of this discussion, they've got some novel
systems and are really focused on these issues.

Thomas Leavitt

On Thu, 2006-06-08 at 07:28 -0700, Matthew Petach wrote:
> (ok, one more set of notes and then off to sit in traffic for an hour on
> the way to work... --Matt)
> 2006.06.06 Power and Cooling panel
> Dan Golding, Tier1 research, moderator
> Hot Time in the Big IDC
> Cooling, Power, and the Data Center
> 3 IDC vendors, 4 hardware vendors
> Michael Laudon, force10
> Jay Park, equinix
> Rob Snevely, Sun
> Josh Snowhorn, terremark
> David Tsiang, cisco
> Brad Turner, juniper
> Brian Young, S&D
> The power and cooling crisis
> internet datacenters are getting full
> most of the slack capacity has been used up
> devices are using more and more power
>  low power density - routers, full sized servers
>  medium power density - 1u servers, switches
>  high power density - blade servers
> Many data centers are full at 70-80% floor space
>  utilized
> North America IDC occupancy is around 50%
>   most sought-after space is around 70%
> full when power and cooling capacity is used up,
> floor space is vacant but can't be used.
> There is a relationship between power and cooling
>  devices are not 100% efficient
>  I^2R losses means that power becomes heat
>   (conservation of energy)
>  heat must be dissipated
> The ability to dissipate heat with normal cooling
>  technologies is hitting the wall
>  need new techniques
> Some quick rules of thumb
> a rack or cabinet is a standard unit of space
>  from 30-40sqft per rack
>  power is measured in watts
>  many facilities do around 80-100w/sqft; at 30sqft
>   per rack, that's about 3kw/rack
> high
> how did we get here?
> what is current situation
> where are we going?
> [dang,  he's flying through his slides!!]
> Hardware engineers
> T-series hardware engineer for Juniper
> CRS-1 hardware
> E-series
> datacenter design issues for Sun,
> there were other hardware vendors who were not
> interested in showing up, these people were brave
> for coming up here!
> Josh snowhorn, IDC planner
> Jay Park, electrial engineer for equinix
> Brian Young, S&D cage design specialist
> What do the IDC vendors feel the current situation
> is in terms of power/cooling, how did we get here?
> Josh--designed datacenters at 100w/sq/ft, more than
> enough for the carriers; the server guys hit 100w/sqft
> in a quarter rack.  you could cannabalize some power
> and cooling, but still ran out of cooling.
> Now spend hundreds of millions to make 200wsqft
> datacenters, or higher.
> Now, to hardware vendors--why are their boxes
> using up so much electricity, putting out so
> much heat?
> What are economics behind increasing density
> and heat load?
> From high-end router space--it's been simple, the
> bandwidth demand has grown faster than the power
> efficiency can keep up with.  In the past, had
> the ability to improve keep up, do power spins about
> every 2 years, half power; but now bandwidth is
> doubling every year, but takes two years to drop
> power in half.  We've been loosing at this game
> for a while, and running out of room on the voltage
> scale; 90nm is down at 1v, can't go much lower,
> since diode drop is at 0.7v; at 65nm, it's still
> at 1v, there's no big hammer anymore for power
> efficiency.  Need to pull some tricks out, but
> may need to do clock gating, may get some 20-30%
> efficiency gains, but not much more that can be
> pulled out of the bag now.
> Newton was right; you can do some tricks, but no
> magic.  Chip multithreading is one area they're
> trying to squeeze more performance out of; don't
> replicate ancillary ASICs for each core.  Also
> can more easily share memory, and nobody has a
> 100% efficient power supply, so you lose some
> power there too.
> More and more getting squeezed in each rack.
> Also a drive on cost; amortizing costs over
> space and capability.
> reducing costs per port is a big driver.
> And customers are pushing for more and more
> density, since the cost of real-estate is getting
> so high, since each square foot costs so high.
> In Ginza, $120/sq ft for space.
> If you go to places where realestate is cheap,
> easier/cheaper to just build really big rooms,
> and let power dissipate more naturally.
> IDC people agree, some cities are just crazy
> in real-estate costs.  But for those in suburban
> areas, cost of real-estate isn't so expensive.
> 3kw per blade server, put a few in a rack, you
> hit nearly 10kw in a rack.  Soon, will need
> direct chilled water in the rack to cool them.
> But chilled water mixed with other colocation
> and lower density cabinets is very challenging
> to build.
> But need to have enclosed space to handle local
> chilled water coolers in localized racks.
> 20 years ago at IBM, nobody wanted chilled water
> in their hardware.  Now, we're running out of
> options.
> Disagree--other ways of handling the challenge;
> how thermally efficient are the rooms in the
> first place, and are there other ways of handling
> heat issues?
> Cables with a cutout in tiles allows air to escape
> in areas that don't provide cooling.
> Josh notes the diversity between carriers at 40w/sq/ft
> vs hosting providers at 400w/sq/ft is making engineering
> decisions challenging.
> It's not about power really anymore, we can get power,
> it's about cooling now.
> Dealing with space in wrong terms--watts/sq ft, vs
> requirements of each rack.  Charge customers based
> on the cooling requirements?
> If you try to cool 15kw per cabinet, you still have
> limits of how many cfm you can move through a given
> space.  At some point, the air flow vertically through
> the rack starts to starve.
> What about a dual push-pull air system that pushes from
> the bottom and pulls from the top.
> Q: Randy, IIJ, question from the audience.  He'll put as
> hot stuff in there as he can cool, because he wants
> the power, that's life.
> Problem is cooling; over 3kw/4kw over current level,
> the wind tunnel effect gets painful.
> the option of putting water in the cabinets is a
> dealbreaker for many people.
> Fact is, most facilities can't even handle 3kw per
> square meter; any build that can't meet that today
> is unrealistic.
> That's 300+ w per sq. ft.
> Josh has some cabinets at NOTA; Akamai is at 386w/sq/ft,
> they can cool it with huge hot aisle behind it, and
> around carriers at 40w/sq/ft.
> Those are the densest cabinets they have.
> IDCs need to build them and charge a realistic amount;
> people will burn as hot as possible, since they need
> to move more and more data.
> Raise plenums higher, move more air, air coming up
> side of rack and across.
> Currently, equinix is building 4kw per cabinet,
> planning for that in 2007.  A cabinet is about
> 2sq meters, so still not at the density Randy's
> looking for.
> Starting to separate high density users from
> medium and low density users.
> Q: GNI, Derek ? datacenter in SF, 1008 IBM blade
> servers, 2500 sq ft, ping pong table, soda
> machine in surrounding areas.
> need 2500w/sq/ft to deliver the same cabinet space.
> 20kw per cabinet is what they can deliver in
> cabinets.
> He's paying for 100 cabinets, have 12 installed.
> he's still netting efficiency for it.  still gets
> 3% better efficiency, still beats 84 1u pizza box
> servers.  If IDCs could keep up with that, could
> keep physical space requirements more reasonable.
> The costs are exponential for more density.
> Up to a year lead time for 2MW generators, we're
> pushing the envelope on that.  It is an exponential
> increase.
> Budget trauma when those costs get passed on.
> Let the demand stimulate ingenuity.
> The internet industry in general is short sighted.
> 22 million blade servers installed, where they will
> be located.
> Q: BillNorton.  one other dimension.  Life span for
> new datacenters is 10-40 year timeframe, so it's hard
> to adjust midlife to hugely different power and
> cooling demands.
> From a technology point of view, CMOS was last
> great quantum leap, need another great quantum
> leap before we get relief on the cooling footprint.
> Randy is right, the cooling architecture isn't
> optimal.
> CRS1, 20% of power goes to fans to move air past
> convoluted air paths.
> Spreading out the equipment is a mitigation.
> multi-chassis systems will help with that.
> Sun, Juniper, do you see power continuing
> to grow linearly, or flatten out?
> As they go to 40gig or 100gig, the power and
> heat will continue to grow; more gates, more
> heat, more power.  We'll hit a wall soon.
> Cisco and Juniper agree, it's 6/6/06, take
> note, world!
> 20 year shelf life for datacenter, look at where
> they were 20 years ago.  10w/sq/foot back in 1986.
> We've greatly increased the amount of work that
> can be done since that time.   Will machines
> continue to do the same amount of work, or will
> we flatten out on the machine capability curve?
> Element of geographic progression as you double.
> Nobody will ever need more than 20kw per rack!
> (Dan Golding)
> Running into some roadblocks; 100M gate ASICs,
> packing so much power into a single chip, may
> not be linear since can't move that much
> heat out of the chip from the point source into
> the system.
> Q: Patrick from Akamai--mcmurta base, south pole?
> As a brighter light, spot of hope, hottest colo
> in Terremark, finding they don't need more power,
> coming to less power.  Running into getting enough
> spindles, the processors are getting faster and
> drawing less power.  40 amps per rack, used to be
> non-full, now able to fill them more completely.
> not sure if everyone is seeing this, but their
> power consumption is going down.
> Not all doom and gloom, but for next 12 months,
> at least somewhat lucky.
> Chip multicores is a good leap that can help for
> a bit; like Sun's Niagra multicore chips or chip
> multithreading, only about 50% of power is used by
> real processing power, rest is ancillary power.
> Q: Rob Seastrom; BillN danced around the question;
> seen it happen before.  MAE-EAST, mark 3, additional
> liebert challengers tucked in....if one builds a
> datacenter to 4kw/meter^2, how long will that be
> premium space vs no-longer-up-to-par.  Does 20
> year colo life even make sense?  Is the run rate
> steep enough that the number is just one we're
> fooling ourselves over?
> Josh: none of them are running at this density.  it's
> the server density that hurts; carriers aren't as much
> of the pain.
> Separation of infrastructure most likely; Voice, carriers,
> etc. there, and separate datacenters where server floors
> exist.
> 20 years from now--will it be obsolete?  Yes, probably.
> they'll keep doing what they can to help service their
> customers.
> Q: Joel K, from ? -- what's coming in network equipment
> to help cut power?  throttle back linecards that
> aren't running at full bore?
> A: If you make it automatic, service providers would
> consider it; but from the bandwidth demand growth,
> the exponential growth--technology isn't keeping
> up, it's plateauing.  Multithreaded ideas, turn
> off idle logic portions, incremental improvements,
> they're one-shot efforts, won't really help fix
> the slope of the curve.
> May just accept that we need more space, period.
> High speed/low speed fans, only kick up to high
> speed during thermal extremes.  Again, both Cisco
> and Juniper have explored suppressing some gear,
> but customers still want 50ms protect gear response,
> so they can't really shut down.
> Even making heat sinks to move the heat is getting
> challenging!
> Force10 also talked to customers about it; in order
> to do turning off portions and then turn back on,
> incurs latency, buffer some packets, etc; people
> can be sensitive to jitter and latency.
> Pushback has been fairly large from other sources.
> Q: Rick Wesson--to colo vendors--when will heat/BTUs be
> a part of of the charges?  And to server vendors, when
> will heat be a listed component upon which vendors make
> sales?
> A: IDCs don't charge based on heat load.
> Power as proxy for heat right now.
> Cooling overhead is wrapped into cost of sq ft and power;
> costs from utilities have been going up 30% due to
> oil prices going up.
> Might make it easier to add that charge for customers.
> Hardware vendors are certainly seeing power/heat
> limitations in RFPs.
> Building smaller systems with fewer slots to meet those
> RFPs.
> Customers asking for gbits/kw now from network gear.
> Sun notes that total cost of ownership, power may
> cost more to run the server than the cost of the
> server itself.
> Q: Lane Patterson, equinix.  T640, if you redesigned it
> today, what fraction of power would it use today
> compared to past?
> Do they engineer gear to see how much they can pack
> into the same power/heat footprint?
> But customers are also asking for more and more
> capacity, less likely to pay for holding the line
> at same power/heat as previous generation.
> Cisco reiterates that we're running out of tricks;
> we can hold the line for a product generation, but
> after that, we're out of luck.  We may need to
> shift architecture of pops going forward.
> Why not build in cheaper places, and backhaul?
> Q: Jared Mauch, NTT--huge customer demand; no vendors
> are proving interfaces greater than 40gig; not for
> next 3 years at least will there be faster links;
> backhauling from remote locations requires aggregating
> more and more traffic; if link speeds aren't increasing,
> backhaul isn't practical.
> As media companies continue deciding they can
> sell movies, music, and the like online, we may
> start hitting the wall; demand on all sides is
> growing, and we're running out of ways to address
> these challenges.
> Q: Avi Freedman.  Talks to people doing lots of very
> dense disk solutions.  Rackable solutions working
> on high density storage racks using laptop drives.
> 48 disks for you starts to generate a lot of heat;
> thumper product?
> 4u, 196 laptop disk rack unit?  For people who need
> lots of spindles, lots of IOPS.
> A: can't talk about those products, they showed
> up in Jonathan's blog, but don't exist yet.
> There are always going to be limitations, the
> vendor will expect you can run the box in the
> location you're going to put it; that is, box
> has requirements, need to make sure customers
> are installing boxes in areas where the thermal
> issues are being considered.
> Q: phil, rosenthal, ISprime.
> people on the panel are pretty good, not the worst
> offenders.  You need to hit the 1u server people
> where most power is being wasted; Dell 1650 vs
> Dell 1850; processor time, sitting at 90% idle
> on both system for bottom of line servers, do
> we need lower end CPUs on server lines so the
> CPUs won't be idle.
> A: why not use fewer machines, but have them do
> more work each?  Virtualization might help us a
> bit in these areas, where we get more efficient
> use of the servers already in place.
> Equinix notes neutral current dropped a lot,
> people using 208V instead  of 120V, generates
> less heat to the datacenter as well.
> Q: Randy; on left, crew singing I want my p2p.
> will always have max heat in the rack; servers and
> router vendors will keep working as hard as they
> can to do what they can.
> They had to leave 30% of their datacenter empty
> and build a bistro in it because they couldn't
> handle the heat budget.
> The IDC vendors [sorry, missed the comment]
> Q: Tim Elisio, new metric?
> To what extent is standardization, like using
> larger, more efficient ower supplies, or more
> efficient fans, cooling systems, etc. helping?
> A: IEEE meetings talking about some standardization,
> get some savings; the economies of scale helps make
> more efficient products on the standardized products.
> Telecom/router industry is working to old standards;
> may need to re-think what airflow standards are,
> for example.
> More dollars in a particular area helps push
> research and development in that direction.
> Juniper notes internals can be optimized, but
> the external plant and interfaces therein need
> better standardization to get economies of scale.
> Everyone's using multispeed fans, use them when
> you need them.
> 3 orgs, SPEC, ECL forum/EPA (energystar for servers
> and blade servers), and GreenGrid.
> Will see benchmarks coming out; will start asking
> all vendors to start compete on how much work they
> can do per how much power sucked and heat generated.
> Make the hardware vendors compete on how efficiently
> they use power and generate heat; we can then decide
> with dollars on who will win.
> Helps motivate people to optimize on the axis they
> care about.
> But are vendors talking to each other about how they
> can use standardized gear and standardized facilities
> designs more efficiently?
> ASHRAY?, heating/refrigeration group puts specs on how
> machines should be cooled (front to back, etc).
> But vendors don't want to help each other compete
> because that hurts their business.
> Dan Golding, 30 seconds, what would each person like
> to see the folks on other side do to help.
> Brian: asked vendors to have more efficient power
> supplies, more efficient systems that generate less
> heat for them to dissipate.
> Equinix--challenged by power density; customers
> don't understand, they want to put in smallest cage
> as possible.  Need people to understand heat load
> better!
> Asks customers to use blanks in unused rack space
> to isolate cold and hot aisles.  Too much leakage
> from cold aisles to hot aisles.  Put blanks in!!
> Josh.  Everyone building hot datacenters; would like
> to see vendors come into IDCs, test them in real
> world environments, put them in labs, see how they
> stand up to environment, test glycol taps, water taps,
> etc.  Building servers is faster than building
> datacenters!
> Hardware vendor; PGE, worked with them to measure
> the increased efficiency, blanks DO help!!
> Education, amongst each other and customers.
> Watts per sq ft is crazy, do it on a per rack
> basis, makes it easier for customers to understand
> the limitations.
> Force10, if IDC groups got together, if there was
> a forum or group they could work with; right now,
> everyone has different requirements.  Otherwise,
> always doing multiple tradeoffs, if there were
> a more general consensus, easier to engineer for.
> Cisco--good point, get IDCs and service providers
> to meet with vendors, come up with a next generation
> facility architecture to aim and build for.  Hopefully
> make cooling and airflow easier, reduce the amount
> of power used by fans.
> Juniper--sees RFPs from customers, environment specs
> are very diverse; would be good to have common
> standards for customers to aim for; also, update
> some outdated nomeclature, use common terminology.
> Michael, Josh, Rob, Brian, thanks to all the panelists,
> Steve Feldman, we've slipped by 15 minutes, start at 2:15,
> everything will slip thereafter
Thomas Leavitt <[email protected]> - 831-295-3917
Godmoma's Forge, LLC -
- Web and graphic design made spiffy -

Encrypted public key at

Download GnuPG (including for Windows) at to read .asc attachment (encrypted signature)

Attachment: signature.asc
Description: This is a digitally signed message part