Don’t Predict The Future; Plan for It

Kevin Brown VP, Global Data Center Strategy and Technology Schneider Electric Session DFS 2 1

Predicting the Future of Densities…

2

Examples of past projections – all overstated reality

“research shows that the average power density per rack will rise to 12kW in 2014 – twice the average 6kW power density per rack in 2006.” “heat loads per rack are predicted to rise to 40 kW by 2009”

Data centers generally don’t meet their design densities

2 basic reasons density forecasts have been wrong

1. Mixed loads in data centers brings average down Specs typically focus only on servers Average = 3-5 kW/rack

Blades Servers Storage Networking

2. Technology issues and improvements • Lower power servers - HP Moonshot chassis – 360 discrete server cartridges, 1 rack, 9.6kW

• • • Peak versus idle sizing Nameplate versus reality Better power management

Equipment ratings do not reflect actual use What is nameplate?

Worst case power draw across the entire IT system + 25% extra just to be sure + 20% de-rated breakers = nameplate is much higher than the actual power draw

Server Energy Use Varies Peak vs. Idle microprocessors

The Facilities View: Power per U Trend

• • •

Data includes: Pizza box servers Blade servers Power optimized servers

Source: http://datacenterpulse.org/blogs/jan.wiersma/where_rack_density_trend_going

Fully loaded rack configurations – difficult to get to 10kW…

11.3 kW rack with 42 DL360P Gen 8 8 fully loaded Moonshot 1500s

And the focus is on increasing performance per watt

Year Cores Power Average CPU Mark Performance/Watt

Xeon E5520

2010 4 Cores 80 Watts 4588

1

4588/80 = 57 TDP and Performance per Watt Trend

Xeon E5-2670v2

2013 10 Cores 115 Watts 20516

2

20516/115 = 178

Atom Z3770

2013 4 Cores 2 Watts 1328

3

1328/2 = 663 1. http://www.cpubenchmark.net/cpu.php?cpu=Intel+Xeon+E5520+%40+2.27GHz&id=1243 2.

3.

http://www.cpubenchmark.net/cpu.php?cpu=Intel+Xeon+E5-2670+v2+%40+2.50GHz&id=2152&cpuCount=2 http://www.cpubenchmark.net/cpu.php?cpu=Intel+Atom+Z3770+%40+1.46GHz

Economics of density

11

3 key variables drive the density cost equation Bigger, more expensive racks + Greater capacity, more expensive rack power distribution + Less # of racks and less space

At low densities, rack count drives savings; at higher density, rack

$/watt The Density Cost Curve Cost per Watt as Density Increases

3,50 3,00 2,50 2,00 1,50 1,00 0,50 0,00 0 5 10 15 20 25 30

kW/rack

35

CURVE LEVELS OFF

Cost savings diminish after 11 kW/rack 40 45 50 55 60 65 Cost includes: Racks Rack PDUs Rack & clearance space cost Assumes air cooling (no liquid to rack)

Financial model assumptions

• Cost curve includes: • Racks • Rack PDUs • Rack & clearance space cost • Assumptions: • Uniform density across all racks • 120V distribution • Wide rack needed at 8 kW/rack; Wide & deep rack needed at 20 kW/rack • Space cost = $150 / sq ft • No redundancy

Assertion: Design complexity isn’t worth potential savings Cost per Watt as Density Increases

3,50 3,00 2,50 2,00 1,50 1,00 0,50 0,00 0 5 10 15 20 25 30 35 40 45 50 55 60 65 1. Network cabling 2. Power cabling 3. Air flow 4. Thermal ride-through 5. Rack weight 6. Additional breakers 15

Natural breakpoints for density based on circuit breaker sizing point towards 11kW per rack max North America Voltage

120 V 208 V 240 V

20 A

1.9 kW

Single phase 30 A

2.9 kW 3.3 kW 3.8 kW 5.0 kW 5.8 kW

40 A

3.8 kW 6.7 kW 7.7 kW

20 A

5.8 kW 5.8 kW 11.5 kW

Three phase 30 A

8.6 kW 8.6 kW 17.3 kW

40 A

11.5 kW 11.5 kW 23.0 kW

Europe Voltage

230 V

16 A

3.7 kW

Single phase 25 A

5.8 kW

32 A

7.4 kW

16 A

11.0 kW

Three phase 25 A

17.3 kW

32 A

22.1 kW 16

Design specs should identify Peak and Average

● Average density is the average rack density across all racks within the space ● Peak density is the highest rack density within the space that all racks will be designed to support Data supports high degree of confidence with 3-5 kW Data supports high degree of confidence with 11.5 kW 17

Density Summary

• Current technology trends and past history support the assertion that densities will remain below the 10 15 kW average.

• Exception will be for IT specific needs requiring close proximity (HPC) and locations where space is at an extreme premium

….

BUT …what if you are wrong…

19

Provisioning for Uncertainty = Insurance Premium

3 strategies to provision for density uncertainty 1.

Oversize the whole data center 2.

Create oversized PODs 3.

Create oversized Racks Bad Idea….

•

Design strategy should embrace POD deployment

• PODs have a capacity that can be specified PODs should always be contained

Contained hot aisle FRONT Rack Rack Rack Rack REAR Hot aisle FRONT Rack Rack Rack Rack

• PODs can deployed with different specifications within the same data center

Scenario 1 – oversize racks…

Specifying PODs

Scenario 2 – oversize entire POD…

Rack sized to peak target – e.g. 10kW POD PDU sized to average target – e.g. 5kW Rack sized to peak target – e.g. 10kW POD PDU sized to average target – e.g. 10kW < 1% premium

PDUs Coolers

5% premium

IT racks

Provisioning should be viewed as an insurance premium Example: Price Premium per Watt for Power Distribution Sized to Peak *

$0,30 $0,25 $0,20 $0,15 $0,10 $0,05 $ 7,7

* Assuming bulk power and cooling, and cooling distribution sized to 5 kW/rack

11,5

Peak kW

17,3 23,0

Cost premium to accommodate peak of 10kW racks with 5kW average versus “perfect” sizing

Cost of provisioning – different methods 25% premium < 1% premium

Oversize rack

5% premium

Oversize the PODs Oversize the Data Center

Scenario: Design for High but deploy Low density

Designed for… 5 kW per rack, 20 racks per POD – 100 kW 400 kW total data center load But actually deployed… 3 kW per rack, 20 racks per POD – 60 kW 240 kW total data center load Strand 160 kW of bulk power & cooling capacity!

PDUs Coolers IT racks

Contingency plan requires more space

Bulk power & cooling 400 kW If density is deployed less than plan,

spare breaker positions & piping valves

allow for use of oversized power & cooling… Designed for: 5 kW per rack 100 kW per pod PDUs Coolers IT racks Flexibility if density prediction is wrong… Need plan to add space – prefab IT modules?

Scenario: Design for Low but deploy High density

Designed for… 5 kW per rack, 20 racks per POD – 100 kW 400 kW total data center load Bulk power & cooling 400 kW PDUs Coolers IT racks But actually deployed… 8 kW per rack, 20 racks per POD – 160 kW 640 kW total data center load POD 1 – 160 kW POD 2 – 160 kW POD 3 – 80 KW – 10 racks stranded POD 4 – Requires more bulk power and cooling Strand 10 racks and have open 4 space for th POD…

Contingency plan requires more bulk power and cooling

Bulk power & cooling 400 kW Bulk power & cooling 160 kW Need plan to add power and cooling – prefab modules?

Or…consider oversizing a few PODs

• Partition loads – high density pod, low density pod 11.5 kW POD with POD PDU sized for 11.5kW average 5 kW pods with POD pdu sized for 5kW average

Modules also can adjust as you evolve – prefab allows you to move faster

Bulk power & cooling 400 kW Bulk power & cooling 750 kW Bulk power and cooling can be deployed in larger chunks… IT PODs can vary in the future if density changes….

Prefabricated Data Center advantages . . .

Flexibility and Scalability Faster Deployment Predictable Performance

• Defer capital expenses and minimize oversizing.

• Adjust quickly changes in your demand, up or down.

• Build in smaller, more granular phases • Shorten planning cycle and reduce complexity • Simplify build process • Delivery in 16-20 weeks or less (upon design sign-off and receipt of PO) • The density, availability and efficiency you expected • The cost you were promised • Less “hands” involved, less changes to design

. . . vs. traditional data centers

31

Practical Implementation Enforce density policy with Peak and Average specified

• Chances are it will be lower than experts predict (11.5/5kW)

Determine your “insurance premium” for uncertainty

• Oversize rack power distribution, PODs, or whole data center

Develop contingency plans in case you miss

• Usually designing for lower density is easier to manage

Use prefab wherever possible for flexibility

• Prefab allows you to increase speed of deployment, better predictability…

Questions

Kevin Brown [email protected]

401.688.4489

33