Transcript PAC 2.6: Critical Design Elements for Building High

PAC 2.6: Critical Design
Elements for Building
High-Power Density Data
Centers
Randy J. Ortiz
VP, Design & Engineering
1
Data Center World – Certified Vendor Neutral
2
PAC 2.6: Critical Design Elements for Building HighPower Density Data Centers
When it comes to high-power density data centers, all are not created
equal.
As organizations increasingly focus on heavy-duty, transactional workloads
like big data analytics, they’re seeking space that can support upwards of
17kW per rack.
Delivering these super high-power densities while ensuring tolerable
working conditions requires careful planning and significant attention to air
flow management, temperature control and electricity.
3
Topics we will cover
1.
Our ravenous desire for technology and how it has created the
high power density data center environment
2.
Enterprise versus Collocation high power density data centers how do I make it applicable to me?
3.
High power densities impact on data center design
4.
Available cooling solutions to high power densities
5.
The Electrical Issue
4
50th Anniversary of Moore’s Law
The affect that transistors shrink half their size
every two years.
Artificial intelligence will exceed
human intellectual capacity and
control, thus radically changing
civilization in an event called
the singularity.
5
Did you know?
• In 1984, the number of internet devices
was 1000
• In 2015, the number of internet devices
exceed 30 Billion
• By 2020, it will exceed 50 Billion
6
What is the issue with
Faster Processing? – HEAT!
There are two main issues regarding faster processors and those are:
1.
Transmission delays on the chip
2.
Heat build-up on the chip
Transmission delays -As the size of the wires and transistors have gotten smaller over the
years, the time required to change states has gotten smaller, too. But there is some limit -charging and draining the wires takes time. That limit imposes a speed limit on the chip.
HEAT - Every time the transistors in a gate change state, they leak a little electricity. The
faster a chip goes, the more heat it generates.
7
Do you remember this?
Do you still have this?
8
Smaller and More Servers =
Lots of Heat & Power!
High power density computing
is here if you can design for it
9
Enterprise kW / Rack High Density Study
Most data centers average to
be low density with some
high density cabinets only.
Courtesy of Schneider Electric
Most efficient density range 5-8kW
10
Collocation Data Center Power
Density Trends
•
A decade ago, standard densities were 3-5 kW per rack
•
Today, it is not uncommon to see 8-12 kW per rack with some deployments
reaching 15-20 kW per rack
•
Not just Internap, others in industry experiencing the
same trend [1]
•
For this presentation, 10-20 kW is considered as high density
in a collocation facility
11
High Density Benefits
• Lowered Total Cost of Ownership (TCO)
•
Emerson Network Power Study [2] – New Data Center
•
•
•
Compare: 5 kw / rack (400 racks) vs. 20 kW / rack (100 racks) – Low Density vs. High Density
Floor Space: 10, 000 SF vs. 2,500 SF
TCO difference after 5 years is $ 3, 660, 000 less for high density scenario assuming similar
maintenance cost
Categories
Capital Costs
Cooling Capital Costs
Annual Cooling Operating
Costs
Maintenance
400 Racks 100 Racks
Description
@5 kW/rack @20 kW/rack
Building shell, rack, PDU
3,500,000
875,000
CRAC, high density modules, installation
830,000
1,900,000
Cooling energy costs
No Data
Estimated TCO ( End of 5th Year)
946,080
No Data
9,060,400
525,600
No Data
5,403,000
Reduced Space
@ $150200/SF
Reduced # of
PDUs @ $80100K/ PDU
12
High Density Benefits
• Increased Scalability
•
Example: Customer starts up with 6 kW / rack, and after a few years has need
for 18 kW for their equipment
Current
Future
13
High Density Challenges
•
More cooling required in smaller footprint
•
Higher cooling equipment initial costs
•
Stronger weight bearing structures required for heavier racks
•
Response time in the event of a cooling failure
•
•
•
Example: If the chiller pump failure or CRAH fan fails, data center operator will have less than 1
minute for 20 kW / cab regions before temperatures exceed ASHRAE recommended temperatures
Solution: Feed CRAH fans and chiller pumps from UPS
Power distribution
14
Mixed Environment Challenges
•
Not all racks in collocation data center are high density: equipment from the past and
present creates a mixed environment
•
Can have 2 kW per rack and 20 kW per cabinet in same data center space.
•
These servers have different operating requirements (e.g. rack inlet temperatures and air flow rates)
•
From a cost / watt perspective, average densities of 5-8 kW per rack is optimal [3]
• Stranded Capacity (Space, Power & Cooling)
•
Available floor space and racks but no remaining power or cooling
•
Air handlers have remaining cooling capacity but not enough air flow rates to match IT equipment requirement
•
PDU has remaining electrical capacity but RPPs out of breaker positions, caused by low density racks.
• Resource Management (Reducing Stranded Capacity)
•
Quantity and location of air handlers driven based on cabinet densities in each region
•
Proper planning and book-keeping for power distribution to avoid overloading PDUS and to balance phases
•
Build out in modular pods to minimize stranded resources.
15
Cooling Strategies for Mixed Environments
• Blanking Panels & Rack Skirt
• Data Center Zoning
• Vertical Exhaust Ducts (VED)
• Cold or Hot Aisle Containment
• In Row Coolers (IRC)
• Other Strategies
16
Blanking Panels & Rack Skirts
•
Recommended for any kW / rack for proper
air flow management
12 kW / rack with and without blanking panels & rack skirts
17
Data Center Zoning (Underfloor)
Similarly, owner can build out high density cabinets in contained pods (hot and/or cold aisle)
18
High Density Cooling Solutions
• No Raised Floor –
(although it could be used for
routing water pipes)
• Row Cooling
• Hot Aisle Containment
19
High Density Cooling Solutions
• Raised Floor Supply
• Chimney Racks
• Ceiling Plenum Return
20
High Density Cooling Solutions
•
Raised Floor Supply
•
Fully Contained Hot Aisle
•
Ceiling Plenum Return
21
High Density Cooling Solutions
•
Hybrid Cooling
•
Raised Floor Supply
•
Row Cooling
•
Hot Aisle Containment
•
Ceiling Plenum Return
22
High Density Cooling Solutions
•
Raised Floor Supply
•
Rear Door Heat Exchangers
•
Ceiling Plenum Return
23
High Density Cooling Solutions
Direct Server Liquid Cooling
24
Other Strategies:
•
Minimize underfloor obstruction by planning route of conduit and piping to limit the
maximum obstruction in any given area
•
Strict hot aisle –cold aisle configuration with containment
when necessary
•
Balance tile flow rates to match required rack flow rates:
manually or use tile with automatic dampers
•
Overhead coolers: similar to in row coolers but do not occupy
floor space.
•
Direct Liquid Cooling – provide water based cooling directly to
chip level
25
The Electrical Issue of High Power Density
•
Proper management of power
cables and electrical distribution
system
•
Management of air flow due to # of
conduits if placed under floor
•
Flexible distribution design plan is
needed
•
•
Number of remote power
panels and their location
Cost
26
Power densities in an average Enterprise Data Center
•
The very low loads are mainly rack enclosures with wiring patch panels, switches, and hubs
•
Loads in the 1 kW range are mainly sparsely populated rack enclosures
•
Loads in the 2-3 kW range are mainly rack enclosures that are populated with typical
equipment but with significant unfilled rack space
•
Loads in the 5 kW range are partially loaded with 1U servers, or contain a mix of
technologies
•
Loads in the 7 kW + range are extremely rare but, according to customers, are going to
become more common with the recent density increases resulting from server technology
advancements
•
Collocation Facilities must provide for all the above and in INAP’s case up to 20kW/
Cabinet
27
The Electrical Issue of High Power Density
Checklist
1. Know your equipment power requirement (Voltage, Amps, Phase, Primary,
Redundant) needs to allow electrical design planning. If you’re a collocation
provider you must prepare for all types of circuits
2. Be sure your team understands how to manage circuits and determining
overloading situations before they are installed
3. Pre-plan your electrical architecture with your rack plan layout to manage conduit
and circuit locations
4. Design your system to be flexible to avoid stranded capacity, installation cost
sticker shock and risk reduction
28
The Electrical Issue of High Power Density
Infrastructure deployment needs to consider:
•
•
•
Sold / reserved Customer power capacity
Actual capacity being drawn by the racks
Ensuring enough circuit / pole positions are available
Infrastructure topology should:
•
•
•
Minimize upfront / Day 1 installation costs
Avoid prematurely installing or stranding infrastructure
Scale without significant cost premium to accommodate more racks & more power capacity
Customer load profiles at INAP data centers are trending towards an INCREASE in kW /
Rack or Power Density (w/sqft)
•
•
Increased risk of inadvertently overloading the infrastructure
Increased risk of Customer power capacity demand out-pacing the ability to deploy infrastructure
29
The Electrical Issue
We want to ensure that our power infrastructure does not have
STRANDED capacity
•
The generators, switchgear, UPS are the most costly components; we want the capacity of
those to be “fully COMMITTED”
The Customer power circuit demands vary throughout the Data
Hall
•
Each circuit requires a circuit breaker that uses “pole positions” within an RPP panel
•
To ensure we do not strand capacity, we need enough circuits to handle “low density” racks
•
“High density” racks require fewer pole positions and fewer RPPs
30
The Electrical Issue
1)
2)
There are different SIZES of Circuits (capacity) that can be provided:

The table below indicates a number of different breaker configurations and the resulting kW

The most popular circuits are 208V/30A, 2-pole (5kW) & 208V/30A, 3-pole (8.6kW)

Although the normal power draw across these circuits may be lower, there is nothing to prevent a customer from drawing up to
the maximum circuit load at any point in time
The sold or COMMITTED capacity is always significantly higher than the ACTUAL capacity

If COMMITTED capacity EXCEEDS the infrastructure capacity then it is OVERSUBSCRIBED

Even if you monitor the actual power consumption, a customer may increase their power draw without warning

There is no means to automatically shed load from the RPP, PDU, or UPS
31
The following slides indicate a phased deployment approach that allows the infrastructure to be
progressively installed to meet the customer needs
1.
2.
Each Data Hall can be sub-divided into 3-zones (upper / middle / lower)
•
Initially install ONE UPS and 3 PDU pairs (6 PDUs) for a data hall
•
Capacity across the total datacenter is up to 1200kW
As customer racks are deployed in a given zone install additional RPPs to support LOW power
density
•
Each zone initially provisioned with up to 500kW of capacity
•
Up to 10 RPPs (4 panel with dual 400A inputs, RPP Option 3) per zone (5 A-side, 5 B-side)
means there will be sufficient circuit capacity
•
RPPs to be floor mounted to reduce length of branch circuits
3.
Monitor the overall power consumption and install an additional UPS as required
4.
As the power density in a given zone increases deploy additional PDUs and RPPs to serve that
increase in load
•
Additional RPPs to be wall-mounted (2 panel with single 400A input, RPP Option 1) for
easier retrofit installation
32
The Electrical Issue
4-Panel
RPP (Aside)
4-Panel
RPP (Bside)
PDU Pair
(Fed by
UPS 1)
33
The Electrical Issue
2-Panel
RPP (Aside)
PDU Pair
(Fed by
UPS 2)
34
3 Key Things You Have Learned During
this Session
1.
The High Power Density option is available, but for the average Enterprise it
remains a small footprint. Internap provides this option for all customers
2.
There are various types of cooling solutions dependent upon the actual
density that needs to be addressed
3.
Electrical distribution in a high power density data center requires pre-design
and proper circuit management to
avoid issues
35
References
1. Clark, J. (2013, October 24). Raising Data Center Power Density. Retrieved March 19, 2015, from http://www.datacenterjournal.com/it/raising-datacenter-power-density/
2. High Density = Lower Cost ( Efficiency Without Compromise E-Book Series, Chapter 2). (n.d.). Retrieved March 19, 2015, from
http://www.emersonnetworkpower.com/documentation/en-us/brands/liebert/documents/white papers/emerson network power higher density equals
lower cost.pdf
3. Brown, K., Torell, W., & Avelar, V. (n.d.). Retrieved March 19, 2015, from http://www.apcmedia.com/salestools/VAVR-8B3VJQ/VAVR8B3VJQ_R0_EN.pdf?sdirect=true
36
Thank you
Randy J. Ortiz
VP, Design & Engineering
37