Build vs. Buy: Internet Datacenter Operations Research INTAP March 6, 2001 Tokyo

Agenda • • • • My Background and Who is Equinix? – 15 minutes Recent Internet Operations Research – 20 minutes “Build vs.Buy Internet Datacenter” White Paper – 40 minutes Future: Where our Industry is heading – 5 minutes

NSFNET to Post-NSFNET Era Internet

Merit NWNet OarNet NyserNet

NSFNET (1987-1994) Regional Tech Meetings

BARRNet NEARNet NSFNET

–Performance & Operations

Regional Regional

–Reporting & Coordination

SDSC NCAR Sesqui Regional

–Routing & Network Planning

OarNet Merit NWNet NyserNet

Privatization (1994-Now) North American Network Operators Group (NANOG) Meetings

MY JOB: 5 Years as Chair of NANOG Regional Regional 12 Years in Internet…No Internet Preso… ANS UUNet Sprint MCI AT&T Regional Regional

Internet Statistics • 87.5% of all Internet statistics are made up.

Made Up Real

Equinix, Inc.

• Scale – size of Internet Business Exchanges, how many, where they are, future plans • “Virtual Fly Through” Tour of the IBX - 6 minutes

• • • Industry Leading Security

5 security checkpoints to customer equipment State of the art video imaging and archiving On site security personnel 24x365

IP/Carrier Knowledgeable Staff • •

24x365 availability

IBX™ Centers Customer Friendly Spaces – Conducting Business & Equipment • Preparation

Designed from customer experience

• • •

Ergonomic work spaces

Fully Redundant Backup Power

Built to 3x power capacity 3 levels of power backup Customer trouble shooting

•

Forward looking design for growing industry

•

Smart Hands

IBX™ Growth

Sites currently open or Under Development

Equinix Internet Business Exchange Model • Infrastructure • Players • Neutrality… Content Providers

Internet Service Providers

Neutral Internet Business Exchange Model

Carriers

Carrier Internet Exchange Model • Infrastructure • Players Non Carrier-Neutral No Content Support Economic Distortions

Internet Service Providers

Carrier Operated Internet Exchange

Carrier Operator

Job Title: Equinix Co-Founder & Chief Technical Liaison • So, my job is to take friends out for lunches and dinners… • Internet Operations Research & • Write White Papers • Summary: What did I learn?

No Internet Operations Documentation To make things more difficult, subtle language differences!

The Internet Operators: All about relationships

My Job: White Papers New Job Title: Equinix Co-Founder & Chief Technical Liaison

Document Internet Operations Research

– “Interconnection Strategies for ISPs” – “Internet Service Providers and Peering” – “Peering Case Studies” – “Peering Simulation Game” – “Build vs. Buy Internet Datacenter”  today

White Paper: Interconnection Strategies for ISPs Direct circuits replaced with: •OC-12 into Exchange •Fiber cross connects

Cost Comparison of Interconnection Strategies S G

$400,000

S G S G S G OC-12 OC-12 S G

$350,000 Cost Drivers: $300,000 •Big OC-12 Direct Circuit •Cheap Fiber Cross Connects $150,000 •Aggregation Efficiency over $50,000 OC-12 (2:1,3:1) •Rack Fees $-

# of participants U A C U A C U A C U A C OC-12 OC-12 OC-12 U A C

Exchange Model Cost Formula: ExchangeModelCost=BigDirectCircuit()+(n-1)*x/2+RackFees() OC-12 

$23,000/mo XC=$200/mo Rack=$1500/mo

So does the Exchange-based model make sense?

White Paper: Internet Service Providers Facilities Allocation 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% Facility becomes widely known Critical mass is achieved Bulk selling of rack 10-packs Cross connect revenue grows rapidly First Carrier First ISP(s) First CP(s) Large Facility Scaling Secondary Carrier(s) Multiple ISPs provide CP redundancy “The Startup Hump” & Peering Facility fills

Peering

Time ISP A Transit $$ Transit ISP ISP B

Peering

Transit $$

WestNet USNet EastNet Routing Table

White Paper: Case Study: Peering in the Bay Area

Transit vs. Peering Alternatives

$3,500 $3,000 $2,500 $2,000 $1,500 $1,000 $500 $0 TRANSIT IBXCustsGig XODS3 XOOC3 XOOC12 Telseon GigE XOOC48 MAE-East OC-3 MAE-East OC-12

Number of 10 Mbps

White Paper: The Peering Simulation Game Transit Provider X A A A X A IXW Y C C C C A A C Transit Provider Y C A A IXN B B B A rolls 3, Pays Transit on others squares (3*$1000) Can get to IXW, IXS Pays Transit on others squares (13*$1000) Pays Transit on others squares (17*$1000) Receives revenue on 4 squares (7*$2000) Receives revenue on 4 squares (2*$2000) $14,000 - $11,000 = $3,000 IXE X Y YES: -$1,000 + both lose a turn Neither has to pay transit to each other!

D IXS C

White Paper: Build Vs. Buy Internet DataCenter • Modeling: • When does it make sense to build your own?

• To Outsource?

• Issues to consider • Economies of Scale?

• Multiple Business Models

Service Level Agreements in Principle • Guarantee a higher level quality • Define Measurement Criteria for adhering to higher quality service Level • Meeting Performance Level critical to success of business • Failure to meet Service Levels penalty significantly worse than cost of service – eBay: $64,000/minute of downtime

SLAs in Practice • No SLA Pricing – ISPPrice NO SLA =X • SLA Pricing – ProbabilityOfMissingSLA=p – ISPPrice SLA1 =X*(1+p) • Network does not typically change to meet new criteria • CIO Security Audit Story • Equinix Switch SLA: 99.999% Uptime

SLA-Datacenters • State-of-the-art Datacenters • Infrastructure Failure unlikely • SLAs rare, but… • Typically Limited to free month service • (Insufficient to potential loss customer faces) • But what is an Internet Datacenter?

• Include hosting? Include Internet Service? Include Caching, Content Distribution?

• Models help Define and Frame the Problem…

SLA Datacenter Parameters • If there were Internet Datacenter SLAs • Parameters: – Installation Lead Time – Service Availability – 24/7 Customer Support – Provisioning/Responsiveness • Independent of Transport and Internet Service Issues.

Creating the Build vs. Buy Internet Datacenter Whitepaper • Create a community white paper on issues associated with the building & scaling of Internet Datacenters • Identify the issues that folks should consider when making the build vs. buy decision Today: • Demonstrate the document & models to date and get feedback NOTE WELL: • EARLY STAGE OF BVB WHITE PAPER (v0.4)!

• This Model may be of help to you in your SLA Discussions

Toured 32 (now) data centers

Some with very Secure Fiber Access… Source: Technology Center of the Americas Tour

Some with less secure fiber entry… Source: Don’t Ask.

Some Very Secure… http://www.thebunker.net

Some Cat 5 Hurricane proof…

Some not quite cat proof.

Loading Dock

Great religious debate over ties Velcro vs.

Plastic cable ties vs.

String

Time to white paper what I’ve seen & Facilitate BOF Discussion • Datacenter BOF I NANOG Oct ’98 Atlanta • Build vs. Buy White Paper • Internet Datacenter Model – Definition of Internet Datacenter • Focused discussion of Each Component of the Model

Dependence Relationship Internet Datacenter Model Increased Value

Value

Customer Business Model

Dependence

Internet Datacenter Model Example IXP Value Dependence

Ultimate Goal Size • Build Financial models that allow an ISP to say: • “Given our size requirements, it makes sense to build our own.” Or • “Given our size requirements, it probably makes sense to outsource this to someone who can realize economies of scale.” + + = Unit cost Buy Build

Flavor of the Real Estate Model • Place to put stuff • Proximity to Fiber • Space Size • Utility Power • Facilities Yard Scaling of Datacenter Real Estate?

Increased Value Relationship

Scaling of the Real Estate Model $/sq. ft.

Premium Real Estate Market (Tokyo, NYC,…) Real Estate Market (everywhere else) # Sq. Ft.

Real Estate Size is Important • Choose Too Small: Run out of space, lose economies of scale • Choose Too Big: Cost per Rack very high Vs.

• Right of First Refusal (some Internet Datacenters allow this) • Expandability of Space

Commitment to DataCenter • 1 yr contract Vs.

• 10 year lease Time to Market Issue – 6 – 9 months to build

Hidden Charges • Meet Me Rooms • Required products: – Must buy port on switch or certain volume of BW • Cross Connect Fees

Size & Finding Real Estate • 100,000 sq ft in downtown DC?

• 10,000 sq ft in downtown DC • Vs.

• Fiber Miles to Internet Datacenter

Flavor of the Infrastructure Model • Heating Venting Air Conditioning (HVAC) • Security Infrastructure • Power Infrastructure • Fire Suppression

Heating, Venting, Air Conditioning (HVAC) Infrastructure • Controlled Humidity ~45% • Too much Humidity: System Failure • Too Little Humidity: Static Electricity • Air Temperature:~72 (IBM: 65-85) • Solutions: • DX Systems • Air Cooled • Water-Based • How do these Scale?

HVAC Cost

HVAC System Cost: Economies of Scale

HVAC Scaling Direct Expansion $1,000/ton Air Cooled $478/ton Chilled Water System $250/ton DX Systems: 25-300 tons of Cooling ~ 10K sq ft or ~150-200 racks Terramark Outsource of HVAC to FPL… # Racks

Crossover point where Chilled Water lowest per-rack solution

Cost Per Rack

$50,000 $45,000 $40,000 $35,000 $30,000 $25,000 $20,000 $15,000 $10,000 $5,000 $ DX System Air Cooled Water Cooled

Num ber of 10 Rack Bundles

Trend? Outsourcing of HVAC FPL HVAC (Miami FL) Water Chiller NOTA: 3 Blocks Away Source: NAP of the Americas Tour

Security Infrastructure • 24x7 Manned • Access: Biometrics, card keys • Video Surveillance Sliding Glass

Power Infrastructure • Sun Netra 1U=90Watts – 40U Rack=3600 Watts!

• Power Source – Power Grid (dual feeds) – Co-Generation – UPS/CPS/Generators • Energy Storage – UPS Batteries – Flywheel (no UPS) – Fuel Cells (service) • Monitoring Issues • Redundancy Issues • TBD: Scaling Issues

Fuel Cells for Generators

Fire Suppression Infrastructure • Two Dominant Systems: • VESPA air sampling • Dry pipe, pre action water system vs. FM-200 • Smoke & Heat Required to activate region

Infrastructure Operations • Operations of HVAC, Security, Power systems • N+1 Redundancy

Internet Datacenter Model

Transport Service Model • Dark Fiber Provider(s), Carrier(s), Aggregators (like Universal Access), etc.

• Business Model?

– Owned by Carrier? Will others come in?

– Not owned by Carrier? Will Carriers come in?

• What type of services available?

• How long to Provision?

• Big Problem: Keeping up with BW Demand

Internet Datacenter Model

Internet Service Model • ISP(s), Aggregator(s) like InterNAP?

• Business Model – Owned by ISP? Will others come in?

– Not owned by ISP? Will others come in?

• Capacity Issues • Latency Issues • Interconnection Strategies Allowed?

Internet Datacenter Model

Content Provider Model • Hoster Focus: – Installing New Servers, Provisioning Host Services, – Managing Server Farm, SLAs, Tech Support, 24/7 NOC, Back Office Solutions • Hoster owns Datacenter?

– Pro: Can buy Transit Services – Con: Expensive w/Bandwidth Hungry Apps – Con: Shared bandwidth issues (Yahoo! attack) – Con: Local Loop Expensive • Business Model Content Distributor(s), Hoster(s)

Internet Datacenter Model

Future of the IXP?

DSL Access Providers Wireless Access Svc Dial-in Modem Operators USENET Svcs E-Mail Svcs WebHost Svcs Virtual ISP Search Engines Cache/Content Distribution Sparing

Service Providers

Internet Service Provider Internet Service Provider Internet Internet Service Service Provider Provider Carrier Carrier Carrier Carrier

MarketPlace: Stratification

of component operations, fast creation & provisioning of packaged operations services

Implication:

IXP 

Content Sources

,  bandwidth demands

Future SLA Challenges 1.

2.

3.

Bundling of Internet Operations Services… Into Cohesive SLA-able services Notion: Trickle-down SLA Bundle Participant Operations Agreements Financial Agreements

Example:

Bundle Internet DataCenter, Transport, Internet, MSP. What happens when

Transport

service fails? Who covers the cost of the SLA?

BvB Document Availability • All references are in the BvB white paper – Hyperlinks work in electronic version of document – Early Stage of document: Comments Welcome!

• Send e-mail to [email protected]

Subject: Build vs. Buy White Paper

Thanks to • Sean Donelan (Equinix) • Jason Colangelo (APC) • Stephen Stuart/Phil Steffora (MFN) • All Colo/ISPs tour facilitators • Friends at Exodus, Frontier Global Center, Level3, etc.

• Daniel Golding (NetRail) • Kenney Huang (AsiaInfra) • Sandip Gupta (Ensim) • Jay Adelson (Equinix) • Josh Horn (NOTA) • Justin Newton (NetZero) • Rob Glenn (Sun) • Everyone here! Please e mail feedback.