Transcript Content Peering William B. Norton Co-Founder and Chief Technical Liaison Content Peering Forum September 18, 2002 © Equinix 2002

Content Peering
William B. Norton
Co-Founder and Chief Technical Liaison
Content Peering Forum
September 18, 2002
© Equinix 2002
Internet Researcher
•
1.
2.
3.
4.
5.
White Papers: Focused studies on Peering
Identify Relevant Internet Operations Topic
Speak with prominent Peering Coordinators
Write/Evolve Draft White Paper
Walk Peering Coordinators through paper
Goto Step 3
Most widely read WP: “Internet Service Providers and Peering”
Peering Research
• Two-Year Study w/~200 Peering Coordinators
–
–
–
–
–
–
What is Peering?
How do you determine who to peer with?
When does Peering make sense?
When does Peering NOT make sense?
What is the Process of Peering?
How is Peering implemented?
Goal: Document Internet Operations Practices
 I will share with you today The Motives, The
Methods, The Mindset of the Peering Coordinator
Summary Findings of Peering Research….
No Internet Operations Documentation
To make things more difficult, subtle language differences!
Peering is a game of relationships
Level Setting Talk…
• Content Peering is…
– Content-Heavy Companies leveraging Simple
Internet Peering, Peering as ISPs do.
– A Routing Optimization (direct vs. through an
Intermediary)
• Not…
– Content Peering Alliance
(see http://www.content-peering.org/peering.html)
– Or any end-to-end Content Alliance
To this end I will share…
Agenda
WP: “Internet Service Providers and Peering”
1. Definitions
•
Transit & Peering Terminology and Financial Cost Models
2. Introduction to the Peering Breakeven Analysis
•
Specific Examples (San Jose Peering) leading to
Generalizations
3. Peering Process
•
3 stages of Peering
4. Top 5 Reasons Not to Peer
5. Peering Resources Available to you
Peering Makes Sense for Large Scale Content Players
Definitions
• Gives us tools (lexicon) to facilitate discussion
• Market Confusion over misuse of terms:
Transit
Peering
Transport
Tier 1 ISP
Effective Peering Bandwidth
• Terminology critical to have meaningful
discussion on Peering
Defs: ISPs
Understand that…
1. Def: The Internet is a network of networks
2. Def: Internet Service Providers sell access to the
Internet
3. Internet Service Providers must themselves
connect to the Internet.
•
How do they connect to the rest of the Internet?
Definition of Transit
Def: Transit is the business
relationship whereby one ISP
announces (usually sells)
reachability to the *entire*
Internet to a customer.
2) Reachability
Announcement
ISP A
1) ISP A buys Transit Service
Upstream
Upstream
Upstream
Upstream
Transit
Transit
Transit
Transit
Providers
Provider
Provider
Provider
3) Traffic Flows
•$$$ ? Typically usage-based
•Pricing ’01: $100-$1200/Mbps
•Volume based on 95th Percentile measure
•Transit is Simple, Convenient:
• Upstream handles the delivery of packets
to the Internet by some means
Usage: “I’m purchasing transit from Level 3.”
I
N
T
E
R
N
E
T
N
E
T
W
O
R
K
S
Cost of Transit Traffic Exchange
Transit Cost Model
$350
$300
$/Mbps
$250
$200
Transit Cost per Mbps
$150
$100
$50
# Mbps Exchanged
43
40
37
34
31
28
25
22
19
16
13
10
7
4
1
$0
Transit Costs
Mbps
$/Mbps
1-3Mbps
$300
4-10Mbps
$220
10-30Mbps
$205
30-50Mbps
$180
50-75Mbps
$146
75-100Mbps
$129
100-300Mbps
$112
300-1000Mbps
$105
2001 Pricing
Sampling
Range: $100-$1200/Mbps
Blended
Avg: ~$200/Mbps
wholesale Transit
My
Financial
used $388/Mbps 95th Percentile
Source:
2002
Survey Models
Range: $50-$1200/Mbps
Isn’t Transit Good Enough?
Traffic Growth
Streaming Media
Yahoo! Broadcast: 100,000+ concurrent unicast streams
15 million streaming hrs/mo, 1300 Live events/day
Streaming:
Broadcast
U
IG
M
Telephony
B
Video
P
Multimedia
$$$
T
Interactive
Gaming
S
Content Heavy ISP
SPOT Events
V
T
R
E
$$$
A
Access Heavy ISP
M
S
56k
384k
1.5m
Access
AOL+ DSL: 1,000/day
Roadrunner Cable Modems: 1M subscribers
Two main motivations for Peering…
KaZaa!!!!
Why Peer? 2 Motivations for Peering
1. Financial: Reduce load on expensive Transit service
Traffic src/dest
Measure vs Intuit
Usage-based Billing
2. Engineering: Lower latency
Transit
$$$
ISP A
Seek transport
Interconnection
$
x
Transit ISP
ISP B
1st Stage of Peering:
Top 10 destination ISP list
Transit
$$$
Def: Peering…
Definition of Peering
Def: Peering is the business relationship whereby ISPs
reciprocally announce reachability to each others’ transit customers.
Peering
Peering
WestNet
USNet
Routing
Tables
EastNet
Transit
•Peering is *not* a transitive relationship
•Peering *does not* provide access to the entire Internet
Usage: “I buy transit from UUNet and Peer with EastNet, WestNet.”
Cost of Peering: Def. Transport+Port
The Costs of Peering
Peering_ Costs  t  p  r  R
where:
t  Monthly_ Transport _ Costs _ Into _ IX
p  Monthly_ IX _ Port _ Fees
r  Monthly_ Rack _ Fees
R  Router_(Equipment_ Costs)
Observation: Transport Prices have dropped like a rock.
Observation: New Router prices have dropped like a rock.
Observation: Used Router Market is also very healthy (cheap).
Graphically…
Ethernet-Based Peering Model
Flat Monthly Fees vs. Metered Monthly Fee
When does Peering
make sense? (SJ Mkt)
San Jose Market Prices for Peering
Peering $
Transit $$$
1) Transport into
Exchange
OC3@$2500/mo
ISP A
2) Rack Space at
Exchange Point
For Router
½ rack $2500/mo
3) Switch Port on
Public Peering Fabric
GigE
4) Cisco 7500 $2000/mo
Total Cost of Peering
$7000/month
R
Transit ISP
X
R
Transit $$$
ISP B
Which is more cost effective?
Peering or Transit? Allocate…
Cost of Traffic Exchange Peering
TRANSIT
vs.
solely
Transit
~ $200/Mbps
Peering $
1 Mbps
ISP A
1 Mbps
Cost of Peering:
$7000/month
R
X
R
Unit Cost of Traffic Exchange
In Peering Relationship:
ISP B
$7000=$7000 per Mbps
1 Mbps
Transit ISP
Cost of Traffic Exchange Peering
TRANSIT
vs.
solely
Transit
~ $200/Mbps
Peering $
2 Mbps
ISP A
2 Mbps
Cost of Peering:
$7000/month
R
X
R
Unit Cost of Traffic Exchange
In Peering Relationship:
ISP B
$7000=$3500 per Mbps
2 Mbps
Transit ISP
Cost of Traffic Exchange Peering
TRANSIT
vs.
solely
Transit
~ $200/Mbps
Peering $
7 Mbps
ISP A
7 Mbps
Cost of Peering:
$7000/month
R
X
R
Unit Cost of Traffic Exchange
In Peering Relationship:
ISP B
$7000=$1000 per Mbps
7 Mbps
Transit ISP
Cost of Traffic Exchange Peering
TRANSIT
vs.
solely
Transit
~ $200/Mbps
Peering $
14 Mbps
ISP A
14 Mbps
Cost of Peering:
$7000/month
R
X
R
Unit Cost of Traffic Exchange
In Peering Relationship:
ISP B
$4000=$500 per Mbps
14 Mbps
Transit ISP
Cost of Traffic Exchange Peering
TRANSIT
vs.
solely
Transit
~ $200/Mbps
Peering $
70 Mbps
ISP A
70 Mbps
Cost of Peering:
$7000/month
R
X
R
Unit Cost of Traffic Exchange
In Peering Relationship:
ISP B
$7000=$100 per Mbps
70 Mbps
Transit ISP
OC-3 Peering vs. Transit in San Jose
Transit Costs
Mbps
$/Mbps
1-3Mbps
$300
4-10Mbps
$220
10-30Mbps
$205
30-50Mbps
$180
50-75Mbps
$146
75-100Mbps
$129
100-300Mbps
$112
300-1000Mbps
$105
OC-3 Peering vs. Transit
$800
$700
$600
$/Mbps
$500
Mbps
Peering
$400
Transit
$300
$200
$100
$0
10
20
30
40
50
60
70
80
90
100
110
120
130
140
150
10
20
30
40
50
60
70
80
90
100
110
120
130
140
150
Peering
$700
$350
$233
$175
$140
$117
$100
$88
$78
$70
$64
$58
$54
$50
$47
# Mbps Exchanged
Generalized…
OC-12 Peering vs. Transit in San Jose
Transit Costs
Mbps
$/Mbps
1-3Mbps
$300
4-10Mbps
$220
10-30Mbps
$205
30-50Mbps
$180
50-75Mbps
$146
75-100Mbps
$129
100-300Mbps
$112
300-1000Mbps
$105
OC-12 Peering vs. Transit
$1,000
$900
$800
$700
Peering
$500
Transit
$400
$300
$200
$100
$15/Mbps
610
590
570
550
530
510
490
470
450
430
410
390
370
350
330
310
290
270
250
230
210
190
170
150
130
110
90
70
50
30
$0
10
$/Mbps
$600
Mbps
50
100
150
200
250
300
350
400
450
500
550
Peering
$190
$95
$63
$48
$38
$32
$27
$24
$21
$19
$17
#Mbps Exchanged
DS-3
OC-3
OC-12
Effective Peering Range
Peering Breakeven Effective Peering Bandwidth
12
42.3
35
145.7
70
584.68
Generalized…
Cost of Traffic Exchange in $/Mbps
Peering Analysis Graph (axis)
Peering Price per Mbps
Peering Breakeven Point
(Peering=Transit)
Transit Price
per Mbps
Transit Cheaper-Peering Cheaper
Effective Peering Range
Min Cost of
Traffic Exchange
(in $/Mbps)
Amount of Traffic Exchanged (in Mbps)
Effective
Peering
Bandwidth (in Mbps)
Cost of Traffic Exchange in $/Mbps
Peering Analysis Graph (EPB)
Peering Price per Mbps
Definition: The Effective Peering Bandwidth
is the maximum bandwidth available for peering,
Peering Breakeven Point
defined as the minimum of the available transport
(Peering=Transit)
bandwidth and the usable bandwidth on the shared
peering fabric.
Transit Price
per Mbps
Transit Cheaper-Peering Cheaper
Effective Peering Range
Min Cost of
Traffic Exchange
(in $/Mbps)
Amount of Traffic Exchanged (in Mbps)
Effective
Peering
Bandwidth (in Mbps)
Cost of Traffic Exchange in $/Mbps
Peering Analysis Graph (minCost)
Peering Price per Mbps
Definition: The Minimum Cost of Traffic Exchange
is the unit cost of traffic exchange
Peering Breakeven Point
when the
(Peering=Transit)
Effective Peering Bandwidth is fully utilized.
Transit Price
per Mbps
Transit Cheaper-Peering Cheaper
Effective Peering Range
Min Cost of
Traffic Exchange
(in $/Mbps)
Amount of Traffic Exchanged (in Mbps)
Effective
Peering
Bandwidth (in Mbps)
Cost of Traffic Exchange in $/Mbps
Peering Analysis Graph (EPR)
PeeringDefinition:
Price per Mbps
The Effective Peering Range (EPR)
is the range in which peering
at an Internet Exchange makes sense (financially),
measured as the range between the
Peering
Peering Breakeven
Breakeven Point
Point and the
(Peering=Transit) Effective Peering Bandwidth.
Transit Price
per Mbps
Transit Cheaper-Peering Cheaper
Effective Peering Range
Min Cost of
Traffic Exchange
(in $/Mbps)
Amount of Traffic Exchanged (in Mbps)
Effective
Peering
Bandwidth (in Mbps)
When does Peering Make Sense?
DS-3
OC-3
OC-12
Effective Peering Range
Peering Breakeven Effective Peering Bandwidth
12
42.3
35
145.7
70
584.68
The 3 Stages of Peering
The 3 Stages of Peering
Interviews with 200 ISP Peering Coordinators
revealed…
3 General Phases of Peering:
1) Identification of Potential Peer – the who
2) Initial Contact and Qualification – the why
3) Implementation Discussions – the how
I. Phase 1: Identification of Peer:
Traffic Engineering Data
Collection and Analysis
Motivations:
• Reduce load on expensive Transit service
Transit
• Traffic src/dest
$$
ISP A
• Measure or Intuit
Seek
interconnection
•
•
•
•
Transit ISP
Usage-based Billing
ISP B
nd
Transit
2 Goal: Lower latency
ResultTop 10 list (see next page) $$
Part of larger business deal
Sample Top 10 Destination List
Internet Service Provider A
AS Number Mbps Destination ISP
6172 24.35 HOME-NET-1
701
Contact
[HOME-NOC-ARIN]
8.90 ALTERNET-AS
[IE8-ARIN]
1668
8.14 AOL-PRIMEHOST
[AOL-NOC-ARIN]
4766
7.08 APNIC-AS-BLOCK
[SA90-ARIN]
3320
5.12 RIPE-ASNBLOCK4
[RIPE-NCC-ARIN]
4.24 BACOM
[EQ-ARIN]
6327
3.90 SHAWFIBER
[IAS-ARIN]
1
3.89 BBNPLANET
[CS15-ARIN]
7018
3.66 ATT-INTERNET4
[JB3310-ARIN]
9318
3.13 APNIC-AS-3-BLOCK
[SA90-ARIN]
5769
2.67 VIDEOTRON
[NAV1-ARIN]
6830
2.30 HCSNET-ASNBLK
[MD205-ARIN]
9277
2.22 APNIC-AS-3-BLOCK
[SA90-ARIN]
2.08 TAMPA2-TWC-5
[JD6-ARIN]
2.05 SprintLink
[SPRINT-NOC-ARIN]
577
10994
1239
M ak e W e
Part of Broad
Business
Relationship?
Dominant Traffic
Flow?
Large new customer
impact?
Yes
Phase 1:
Identification
of Potential
Peer
Traversing
Expensive Transit
Circuit?
Yes
Yes
Will Peering have a
positive affect on my
network?
Yes
Proceed
to Phase
2:
Contact
Peer
Yes
II. Phase 2: Contact &
Qualification, Initial Peering
Discussion
How to make contact with potential peer ISP?
1. E-mail person or peering@<ispdomain>.net
2. Exchange point participant list
3. Tech-c/admin-c from DNS/ASN registries
4. Engineering Forums NANOG, IETF, RIPE, etc.
5. Trade shows: speakers and booth staff
6. Target ISP sales force
7. Target ISP NOC
II. Phase 2: Contact &
Qualification, Initial Peering
Discussion
Once contact is made…
1. Sometimes Mutual NDA
2. Exchange BiLateral Peering Agreement (BLPA)
3. Traffic Data justification shared
One basis: Peering iff PeeringCost < TransitSavings?
4. Requirements Exchange
(e.g. Must be at n Public Peering Points, xMbps, private
peering migration strategy, etc.)
Either Party may walk away…..
If still interested, implementation discussion…
Finding the Right Contact
Larger
Business
Transaction
peering
@ or
personal
contact
Exchange
Point
Contact list
Initial
Contact
Phase 2:
Contact and
Qualification
Sign
NDA,
see
policies
Share
traffic
data,
BLPA
Do both parties find
motivation to continue
peering discussion?
No
Close discussion
Yes
Proceed to
Phase 3:
Implementati
on
Discussion
tech-c or
admin-c
in DNS/
ASN
Registry
Operations
Forum
Trade
Shows
Sales
Force
III. Phase 3: Implementation
Discussions
How to interconnect?
Direct Circuit-based Interconnection
VS.
Exchange-Based Interconnection
Cost Comparison at n=5
OC-12
costDCfn()=(n-1)*C/2
C=OC-3 @ $2,500
n=5
costDC=(4)*$2500/2
costDC=$5,000/mo
S
S
S
S
G
G
G
G
U
U
U
U
A
A
A
A
C
C
C
C
S
OC-12
G
OC-12
OC-12
U
A
OC-12
C
costExchfn()=BDC+(n-1)*x/2+Racks
BDC=OC-12 @ $5,000
n=5, 1 Rack@$1000
costExch=$5,000+(4)(250/2)+$1000 More expensive to use Exchange-Based
Interconnection Strategy at n=5. N>5?
costExch=$6,500/mo
Exchange-based vs. Direct
Circuit Interconnection
Cost Comparison of Interconnection Strategies
$400,000
Direct Circuits Model
$300,000
MUX Big Pipes Model
$250,000
$200,000
Dark Fiber Model
$150,000
$100,000
$50,000
# of participants
64
61
58
55
52
49
46
43
40
37
34
31
28
25
22
19
16
13
10
7
4
$-
1
Monthly Cost of Interconnection
$350,000
9 Exchange Selection Criteria
1.
2.
3.
4.
5.
6.
7.
8.
9.
Telecommunications Access Issues
Deployment Issues (getting in & up)
ISP Current Presences (there yet?)
Operations Issues (restrictions?)
Business Issues (neutrality/alignment)
Cost Issues ($$)
Credibility Issue (backing,attraction) 
Exchange Population (side effect)
Existing vs. Emerging Exchange?
VExchange
Value of the Internet Exchange
VCapacity
Circuit
Demands
T
rickle Down
Content
P
rovider
=
Content Content Cnumber
of
content providers
P
rovider P
rovider
ISP
ISP
I=
numberof
ISPs
ISP
Cost
Of Coming In
(Circuits+
Routers+
StaffTime)
First Carrier(s)
First ISP(s)
First CP(s)
Carrier
The Exchange Startup Hump
Carrier
N=
numberof
carriers
Critical Mass Point (Vexchange=CostExchange)
NParticipants
Top 5 Reasons NOT to Peer
Top 5 Reasons not to Peer
1) Already get Traffic for “free” (through
existing peering relationships)
Transit
$$$
Yahoo!
Peering
$
Transit ISP
EXODUS
AOL
Top 5 Reasons not to Peer
2) Not True Peers
• Traffic inequity
Huge investment in Int’s circuits,
100’s of routers and colo sites,
Staff installs, peering negotiations,
Millions of customers, etc.
Large Global Network Provider
•
•
•
Scale inequity
Not even investments in infrastructure
Form: “I don’t want to haul your traffic
around the globe”
Small
Regional
Player
Top 5 Reasons Not to Peer
3) Lack of Technical
Competence
Troubleshooting network
problems takes longer
when the other ISP
NOC and engineers lack
the technical expertise
during an outage…
Top 5 Reasons Not to Peer
4) Transit Sales Preferred
• We rather sell you transit…“Let me
introduce you to our sales guys”
Top 5 Reasons Not to Peer
5) BGP is Tough
“BGP? No ExpertiseNo measurements
No Justification to hire expertsBGP?”
Simple
Conceptual Hurdle
Primary
Backup Primary
Transit
Transit
$$$
ISP A
Seek transport
Interconnection
$
Backup
Transit
Complex
Conceptual Hurdle
x
Transit ISP
ISP B
Transit
$$$
6: personality
Top 5 Reasons Not To Peer
5+ Personality Clashes:
They don’t understand
each other
and they didn’t like the
interaction
Resources Available to Peering
Coordinators
Resources Available for Peering
Coordinators
• Gigabit Peering Forums
• Other White Papers document Peering
Practices
• Peering Contact Database
Gigabit Peering Forums
Other White Papers
“Interconnection Strategies for ISPs”
“Internet Service Providers and Peering”
“A Business Case for Peering”
“The Art of Peering: The Peering Playbook”
“Do ATM-based Internet Exchange Points make
sense anymore?”
“The Peering Simulation Game”
Freely available from the author: [email protected]
Peering Contact Database
For Peering Coordinators Only
Toss in your Business Card &
Receive a copy of everyone’s Business Cards
Every 6 weeks (or so)
Managed as a community service.
E-mail to [email protected]
(Or give me your business card)
Conclusions
• Language of the Peering Coordinator
– Transit, Peering, Transport
• ISP Peering makes sense if you can offload x
Mbps of traffic to peers…
• The Peering Process includes 3 Phases:
– Identification of Peer
– Contact and Qualification
– Implementation of Peering
• These represent the Baseline Understandings of
the Peering Coordinator
Acknowledgements?
Acknowledgements
For this white paper I’d like to thank a few folks in particular for their review,
insights, and comments on this paper:
Dorian Kim (NTT/Verio), Ingrid Erkman (ICG), Dave McGaugh (ELI),
Eric T. Bell (Time Warner Telecom), Chris Parker (StarNet), Brokaw Price (Yahoo!),
Lane Patterson (Equinix), Jay Adelson (Equinix), Morgan Snyder (Equinix),
John Hardie (Equinix), David Diaz (BellSouth), Joe Wood (Accretive Networks),
Robert Seastrom (inter.net), Kevin Epperson (Level3), Petri Helenius (FICIX),
Scott Sheppard (BellSouth), Ralph Doncaster (iStop.com), Leo Bicknell (ufp.org),
Paul Vixie (vix.com), Ian Somerton and Dave Wodelet (Shaw/BigPipe),
Tony Hain (Cisco), Jeff S. Wheeler (five-elements.com), Cliff Hafen, Dory Liefer,
Shannon Lake (Omnivergent), Nenad Trifunovic (WorldCom), Andre Gironda (eBay),
Jeb Linton (EarthLink), Daniel Golding (SockEye), Peter Moyer (Juniper),
and others that preferred no recognition for their contributions to this paper.
Top 5 Reasons Not To Peer…
Questions?
Do these same Motivations to Peer
apply to Content Companies?
Are the Financial Motivations more compelling than
the Performance Improvement Motivations?