Transcript PPTX
The Netflix Open Connect Network
DKNOG – 21 March 2013
1
Today we are in 22 locations across the world...
(16 Primary, 6 “network extensions”)
2
3
How do we decide what locations to go to?
Where is Netflix presently providing service?
Our service is limited by the content licenses that we can buy
For example, we are in the Nordic countries, the UK, and Ireland but
not France/Germany
Can we interconnect with networks that are relevant?
Going to a peering point that’s all content providers does no good
Can we reduce the cost and improve traffic
management for networks who need to deliver
Netflix traffic to their end users?
4
Netflix in the Nordics
We provide our streaming service to
Denmark
Finland
Norway
Sweden
Iceland and Greenland
Currently co-located at Telecity Stockholm 1 & 2
Stockholm 1 is a network extension from Stockholm 2 due to
space/power constraints
Likely to expand to a second market in the Nordics this
year
5
IXes
Netflix delivers their Nordic traffic from four different IXes
NetNod
NIX
AMS-IX
LINX
The latter two may not seem relevant here, but many
networks have significant capacity built there
6
Standard build - European
A European Netflix Open Connect IX build has:
20 Netflix Open Connect Appliances
Two Juniper MX480 routers
OpenGear console server and Juniper EX2200’s for OOB access
Prewired MTP to LC cabling from Elpeus
A “network extension” Open Connect build has:
Two Juniper EX4550 switches
Two MRV 40 channel passive muxes
OpenGear console server for OOB access
7
Stockholm
8
Network Extension POP
9
Standard Build - US
A US Netflix Open Connect IX build has:
40 Netflix Open Connect Appliances
60 Netflix Open Connect Flash Appliances
Two Juniper MX960 routers
OpenGear console server and Juniper EX2200’s for OOB access
Prewired MTP to LC cabling from Elpeus
A “network extension” Open Connect build has:
Two Juniper EX4550 switches
Two MRV 40 channel passive muxes
OpenGear console server for OOB access
10
11
Converting some POPs to Cisco ASR
Due to concerns with Juniper code delivery and the need
for higher density platforms, we decided in early 2013 to
integrate some Cisco ASR 9K into our network to replace
Juniper MX
In order to deliver our traffic goals (~1T per router), we
need denser cards than the currently shipping Juniper 16
port MPC3’s
12
High Density…
Juniper MX960 w/ MPC3’s can deliver 176 ports per
chassis
A single fabric failure brings the chassis to 3:2 oversubscription
Juniper MX960 w/ MPC4’s can deliver 336 ports per
chassis (oversubscribed)
A single fabric failure (out of 3) brings the chassis to 2:1
oversubscription – unacceptable
Cisco ASR9010 w/ Typhoon-based 36 port cards can
deliver 288 ports per chassis
A single fabric failure brings the chassis to 3:2 oversubscription
13
High Density, continued…
We conservatively need 130 host ports per router
Those 130 ports can deliver 1.13T of traffic
Assuming we deliver 1 bit outbound for every bit inbound
from a host, that puts us at 1.1:1 oversubscribed at peak
3:2 and 2:1 just do not work for us
Juniper MPC4’s also require brand new 12.3 code
What version of code are you running?
14
Cabling a new ASR…
15
The Netflix Open Connect Appliance (OCA)
Developed in response to ISP requests to help scale
Netflix traffic efficiently
Reduces ISP cost by serving Netflix traffic from the local
ISP datacenter, CO or headend, rather than upstream
network interconnects
Speeds up internet access for consumers to all thirdparty internet sites, because Netflix traffic is no longer a
source of middle-mile or backbone congestion
Netflix bears the capital and maintenance costs, not ISP
ISP provides space, power and a network port
An OCA is a component of the Netflix CDN (vs a cache)
16
OCA Hardware
• Space optimized: 4U
high-density storage
• Power optimized for low
power/cooling
requirements (≅500W)
• Redundant 10GE optical
network interfaces
• Redundant power
supplies (AC or DC)
17
Why Deploy a Netflix OCA?
Netflix data is a significant percentage of ISP traffic in the
markets we serve
Serve >80% of Netflix traffic from the local ISP datacenter
Remaining traffic can be served by upstream OCA’s, peering
Saves on transit, transport and other upstream scaling costs
Provided free of charge to participating ISPs
ISPs with > 3 Gbps of Netflix traffic
ISP provides rack space, power, 10 Gbps optical port(s)
OCA Operation
Used exclusively for Netflix content
Completely integrated with the Netflix content delivery system
Greater effectiveness than transparent or proxy caches (>80% with
Netflix OCA, versus 20-50% with other caches)
Efficient content fill mechanisms
Outside peak times (ISP selected time and BW per OCA)
OCA offline during fill, staggered fill recommended.
Fill can source from neighboring, peer or transit OCA’s
~5TB fill per day (i.e ~4Gb/s for ~2.5 hrs)
OCA must be reachable by end users (port 80)
Directing Clients to OCAs
Netflix Control Servers
Broadband ISP
Netflix OCA
3. Client connects to local OCA
4. Local OCA delivers video stream
• User routing is done by
Netflix control servers,
not dependent on client
DNS configuration
• Request is routed to the
nearest available OCA
• Working sets of popular
content deliver up to
>80% network offload
Network Positioning
OCA’s are installed close to aggregations of end users
Netflix works with ISPs to determine proper location
and user-to-OCA mapping
ISP Controls Traffic Flow
End-user netblocks are associated with OCA via BGP advertisement
Netflix honors AS path-length and MEDs allowing ISP to configure OCA
hierarchy and failover
OCA’s only serve content to netblocks provided by ISP
ISPs choose peak throughput each OCA is allowed to generate
The Netflix ISP Speed Index
Published monthly
Helps consumers make an intelligent choice about their
broadband provider
We want our users to have the best experience possible
We also want to help networks deliver Netflix traffic as
efficiently as possible
22
23
A recently enabled network (night of 18/03/13)
No marked increase in bandwidth utilization, but a more stable experience for users
24
Questions?
Additional information:
http://openconnect.netflix.com