Consolidation of a Performance Sensitive Application: Virtualizing Electronic Sports League's Gaming Infrastructure

Breakout Session PO3435

Bob Albers

I/O Usage Architect, Intel 2008-08-26

Sreeram Sammeta

Senior Systems Engineer, Intel

Agenda

• • • •

Server Applications: Can We Virtualize* Everything?

New Technologies Allow Us To Virtualize* More Question The Assumptions ROI Delivered

* In this context “virtualize” means to use a VMM or hypervisor

Can We Virtualize Everything?

Perception: “Non-virtualize-able” Applications

Mission Critical Transaction Latency Sensitive CPU Intensive: Single Thread vs. Multi-Thread Memory Intensive: Size/Throughput/Latency I/O Intensive: Disk/Network; Throughput/Latency Perception: Virtualization impacts all of the above Therefore: These types of applications “can’t be virtualized” because the tradeoffs are too severe

Perception or Reality?

Can We Virtualize Everything?

Electronic Sports League

Largest online gaming community in Europe >844,000 active users * Mission Critical Servers Game Transaction Latency Sensitive Single Thread CPU Intensive Internet

Firewall … Game Servers

“Can’t Be Virtualized” but … ESL motivated to try: sprawl, cost, power, OpEx

* Source: ESL web site 2008-08-12

New Technologies Allow Us To Virtualize More

ESL Proof of Concept (PoC) Overview

Hypothesis: virtualization of gaming servers may be possible if … We use the latest technologies

 Intel® Xeon® Processor 7400-based servers with Intel VMDq NICs  VMware ESX* 3.5U1 & NetQueue

Test network latency impact of virtualization in the Intel lab Private testing @ ESL lab and public testing on the Internet with real ESL members VM 1 vSwitch1 VM n 10 GbE VMDq Force 10 S50N

ESL Network

Firewall Svc Cons vSw0 1 GbE

Prove it!

New Technologies Allow Us To Virtualize More

PoC Server Hardware

Intel® Xeon® Processor 7400 Series (Dunnington) HW and SW compatible with existing Quad-core 7300 based platforms 4 sockets with 6-cores each 4x 1066 MHz Performance boost from 6-core with new 16 MB L3 cache FBD FBD FBD FBD Intel 7300 FBD FBD FBD FBD Energy efficient boost from 45nm high-k process technology ESB2 I/O Bridge Enhanced hardware assist features for virtualization Available September ’08 7400 Series 32 slots

256 GB max 32 GB as tested

Configurable PCI Express*

New Technologies Allow Us To Virtualize More

Importance of I/O Tuned For Virtualization

VM 1

Virtual NIC

VM 2

Virtual NIC

… VM n

Virtual NIC

VMM overhead

  Switching load Interrupt bottleneck

Virtualization Hypervisor NIC LAN

10.0

8.0

6.0

4.0

2.0

Unused I/O capacity 4.0

0.0

w/o VMDq Result: NIC performance can be up to ~60% under utilized

Source: Intel.

Source: Intel. Throughput measures receive side (Rx) I/O performance of 10GbE LAN.

New Technologies Allow Us To Virtualize More

Optimizing I/O For Virtualization

VM 1

Virtual NIC

VM 2

Virtual NIC

… VM n

Virtual NIC

VMDq & NetQueue

  Optimize switching Load balance interrupts

VMware with NetQueue NIC with VMDq LAN

10.0

8.0

6.0

4.0

9.2

9.5

• 2x throughput • Near native 10GbE

4.0

2.0

0.0

w/o VMDq w/ VMDq w/ VMDq Jumbo Frames

Source: Intel.

Tests measure Wire Speed Receive (Rx) Side Performance With VMDq on Intel ® 82598 10 Gigabit Ethernet Controller

New Technologies Allow Us To Virtualize More

PoC Software

 

VMware ESX* 3.5U1

  Virtual Center 2.5

VMDq & NetQueue enabled (16 queues on one 10G port) 1 vCPU per VM 2GB memory per VM

Windows Server* 2003 32-bit CounterStrike* 1.6

Counter Strike 1.6

Windows Server 2003 32 bit VM1 VMware® ESX 3.5 U1 Counter Strike 1.6

Windows Server 2003 32 bit VMn Intel® Xeon® Processor 7400 Series based Server

Enough theory… let’s move on to engineering & testing!

Question The Assumptions

Key Performance Metric: In-Game Transaction Latency

In-Game Latency (mSec) In-game transaction latency = round-trip network latency + game server processing time 25 20 = >20mS typical or 10mS best case on Internet 15 Best LAN =~ 5mSec best case for LAN connected players 10 Best Internet Typ. Internet Player 40-200 bytes UDP 2000 bytes UDP Game Server 5 0

Source: ESL

How much does virtualization increase round trip Network Latency?

Source: ESL observations. Actual performance may vary.

Question The Assumptions

Round-Trip Network Latency Test

Virtualization increases latency; ~double VMDq latency is near native @ 0.13mS for 64-byte packets Negligible since In Game Transaction Latency is 5mS best case 0.30

0.25

0.20

0.15

0.10

0.05

0.00

Netperf 2.4.4 UDP Latency test with 8 parallel streams 64 256 1024 Native VMDq No VMDq Packet Size (bytes)

Source: Intel lab

Network Latency w VMDq is near Native!

Negligible impact to In-Game Latency!

Source: Intel Lab. Performance measured using the netperf benchmark running on Intel® Xeon® processors 7300 (2.93 GHz). Actual performance may vary.

ROI Delivered

ESL Game Testing

Private ESL & Public Internet testing revealed no impact on In-Game Transaction Latency!

Source: ESL Lab. Performance measured using esxtop & power meter with ESL reference s/w stack running on Intel® Xeon® processor 7400 Series based platform. Power measurement is wall power for the entire server as tested including 8 SAS drives. Actual performance may vary.

"Playing on virtualized gameservers running on Intel and VMWare technologies gives professional gamers no disadvantages compared with playing on a non virtualized server. Everything ran smoothly and I did not notice anything unusual. A perfect setup for professional gaming."

—Navid Javadi aka mousesports|Kapio

100 80 60 40 20 0 ESL virtual game servers on Intel® Xeon® Processor 7400 Series based platform 740 720 700 680 660 640 620 24 32 36 40 # of VMs Source: ESL lab

ROI Delivered

ESL Game Testing Results

Server Cores Game server processes Before 1P C2D 2 3 per CPU core VMs per box Game server processes per box Users per box CPU Utilization Power per physical server Consolidation Ratio 6 72 60-80% 350W 18 Annual direct power cost $731,000 per 1000 servers After 4P Xeon 7400 24 3 per VM; 4.5 per CPU core 36 108 1296 75% 710W 1 $83,000 for 56 servers "The new Six-Core Intel® Xeon® 7400 series processors (“Dunnington”) were completely overwhelming in all terms. The Intel Xeon MP servers with Intel VMDq technology enable us to efficiently run our servers with reduced costs and without any negative impacts."

—Bjoern Metzdorf Director Information Technology Electronic Sports League

Source: ESL Lab. Performance measured using esxtop & power meter with reference s/w stack running on Intel® Xeon® processors 7400. Power savings calculated based on 24x7x365 usage, ESL actual power rate & Yahoo $/€ exchange rate as of 2008-08-12. Power numbers and cost savings noted here are for server wall power only. No consideration of cooling power costs and other potential savings is included here. Actual performance and savings may vary.

Power savings of $648,000 per year for each group of 1000 servers converted!

Summary

•

Server Applications: Can We Virtualize Everything?

•

New Technologies Allow Us To Virtualize More

•

Question The Assumptions

•

ROI Delivered

Do you have “non-virtualize-able” applications?

Evaluate the latest technologies.

Prove it!

Additional sources of information on this topic:

Intel Virtualization Developer Community

 http://softwarecommunity.intel.com/articles/eng/1424.htm

Intel Virtualization Technology

 http://www.intel.com/technology/virtualization/index.htm

Intel® Xeon® Processor 7300 Series

 http://www.intel.com/performance/server/xeon_mp/virtualization.htm?iid=SEARCH

More information on Intel® VT for Connectivity: www.intel.com/go/vtc More information on Intel Networking solutions: www.intel.com/network Intel® Virtualization Technology for Connectivity Tech Brief

 http://softwarecommunity.intel.com/isn/downloads/virtualization/pdfs/20137_LAD_VTc_Tech_Brief_r04.pdf

Virtual Machine Device Queues White Paper

 http://www.intel.com/technology/platform-technology/virtualization/VMDq_whitepaper.pdf

VMware ESX Server 3 Configuration Guide – Update 2 for ESX 3.5

 http://www.vmware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_3_server_config.pdf

 Search for NetQueue/VMDq configuration info

Intelligent Queueing Technologies for Virtualization: Intel-VMware white paper

 http://download.intel.com/network/connectivity/products/whitepapers/Intel-VMware_VMDq_wp_May08.pdf

Intel 4-Processor Server System S7000FC4UR

 http://www.intel.com/products/server/systems/4-processor-s7000fc4ur/4-processor-s7000fc4ur-overview.htm

We want to hear from you…..

Join the conversation on Intel’s Open Port: The Server Room Bring your questions to the “Ask an Expert” discussion forum Share Best Practices with IT influencers Engage with Intel on your DataCenter challenges

Contact Info :

Wm. Hank Lea: [email protected]

Josh Hilliker: [email protected]

Check out -- communities.intel.com/community/server

Q&A

Legal Disclaimer

INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL® PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL’S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL® PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. INTEL PRODUCTS ARE NOT INTENDED FOR USE IN MEDICAL, LIFE SAVING, OR LIFE SUSTAINING APPLICATIONS. Intel may make changes to specifications and product descriptions at any time, without notice.

All products, dates, and figures specified are preliminary based on current expectations, and are subject to change without notice.

Intel, processors, chipsets, and desktop boards may contain design defects or errors known as errata, which may cause the product to deviate from published specifications. Current characterized errata are available on request.

Dunnington and other code names featured are used internally within Intel to identify products that are in development and not yet publicly announced for release. Customers, licensees and other third parties are not authorized by Intel to use code names in advertising, promotion or marketing of any product or services and any such use of Intel's internal code names is at the sole risk of the user Performance tests and ratings are measured using specific computer systems and/or components and reflect the approximate performance of Intel products as measured by those tests. Any difference in system hardware or software design or configuration may affect actual performance. Intel, Intel Inside, Intel Xeon and the Intel logo are trademarks of Intel Corporation in the United States and other countries. *Other names and brands may be claimed as the property of others.

Copyright © 2008 Intel Corporation.

Risk Factors

This presentation contains forward-looking statements that involve a number of risks and uncertainties. These statements do not reflect the potential impact of any mergers, acquisitions, divestitures, investments or other similar transactions that may be completed in the future. The information presented is accurate only as of today’s date and will not be updated. In addition to any factors discussed in the presentation, the important factors that could cause actual results to differ materially include the following: Demand could be different from Intel's expectations due to factors including changes in business and economic conditions, including conditions in the credit market that could affect consumer confidence; customer acceptance of Intel’s and competitors’ products; changes in customer order patterns, including order cancellations; and changes in the level of inventory at customers. Intel’s results could be affected by the timing of closing of acquisitions and divestitures. Intel operates in intensely competitive industries that are characterized by a high percentage of costs that are fixed or difficult to reduce in the short term and product demand that is highly variable and difficult to forecast. Revenue and the gross margin percentage are affected by the timing of new Intel product introductions and the demand for and market acceptance of Intel's products; actions taken by Intel's competitors, including product offerings and introductions, marketing programs and pricing pressures and Intel’s response to such actions; Intel’s ability to respond quickly to technological developments and to incorporate new features into its products; and the availability of sufficient supply of components from suppliers to meet demand. The gross margin percentage could vary significantly from expectations based on changes in revenue levels; product mix and pricing; capacity utilization; variations in inventory valuation, including variations related to the timing of qualifying products for sale; excess or obsolete inventory; manufacturing yields; changes in unit costs; impairments of long-lived assets, including manufacturing, assembly/test and intangible assets; and the timing and execution of the manufacturing ramp and associated costs, including start-up costs. Expenses, particularly certain marketing and compensation expenses, vary depending on the level of demand for Intel's products, the level of revenue and profits, and impairments of long-lived assets. Intel is in the midst of a structure and efficiency program that is resulting in several actions that could have an impact on expected expense levels and gross margin. Intel's results could be impacted by adverse economic, social, political and physical/infrastructure conditions in the countries in which Intel, its customers or its suppliers operate, including military conflict and other security risks, natural disasters, infrastructure disruptions, health concerns and fluctuations in currency exchange rates. Intel's results could be affected by adverse effects associated with product defects and errata (deviations from published specifications), and by litigation or regulatory matters involving intellectual property, stockholder, consumer, antitrust and other issues, such as the litigation and regulatory matters described in Intel's SEC reports. A detailed discussion of these and other factors that could affect Intel’s results is included in Intel’s SEC filings, including the report on Form 10-Q for the quarter ended June 28, 2008.