10 Gigabit Networking at TelstraClear Jörg Micheel

NZNOG 2006 – Victoria University Wellington, March 23 rd , 2006 1

Outline • Some Metro Ethernet equipment in use at TelstraClear • Test equipment, testing process • Project: University of Auckland 10 Gigabit Campus Ring • Measurement results and performance figures • Outlook: REANNZ Advanced Network implementation NZNOG 2006 – Victoria University Wellington, March 23 rd , 2006 2

Extreme Networks Black Diamond – metro core switch • Black Diamond 10808 (BD10K) • 22 rack mount units • 1280 Gbps capacity • Up to 48 10-Gigabit ports • Up to 480 10/100/1000 ports • Powerful VLAN, Virtual router Layer2 and Layer3 capabilities • Proprietary EAPS link-protection protocol provides continuity in case of fiber cut • L2/L3 Quality-of-Service • L2/L3 hardware filtering and priority • Jumbo frames at 9212 • High availability, hardware redundancy NZNOG 2006 – Victoria University Wellington, March 23 rd , 2006 3

Extreme Networks Summit X450 – edge switch • X450-24t with 24 ports 10/100/1000 copper, four combined SFP GigE ports • X450-24x with 24 ports 1-GigE SFP, four combined 10/100/1000 ports • Optional dual 10-Gigabit Ethernet uplinks • 1 RU form factor • 160 Gigabits-per-second capacity • 65 million packets-per-second forwarding performance • Coming: X450a with stacking capability • Other features similar to Black Diamond series NZNOG 2006 – Victoria University Wellington, March 23 rd , 2006 4

Network test equipment – EXFO PacketBlazer

• 8510 – wire speed full duplex 10/100/1000 • 8510G – wire speed full duplex 10G LAN/WAN PHY • BERT testing at Layer 1 • Frame testing at Layer 2 • RFC 2544 Performance test suite for network devices • Throughput, back-to-back packets, frame loss and latency, jitter • Detailed QoS analysis • Can extend concept of device testing to (parts of) a network NZNOG 2006 – Victoria University Wellington, March 23 rd , 2006 5

University of Auckland 10 Gigabit campus ring (1)

Requirements: • Resilient infrastructure to interconnect eight University of Auckland campus sites • Provide multiple 1-Gigabit services at Layer 2 between campuses Solution: • Dual-implementation with Summit X450 at six sites within Auckland metropolitan domain • Dark fiber between the sites • Supports multiple unconstraint 1-Gigabit VLAN services with QoS up to the maximum 10 Gigabit backbone capacity • Resiliency via Ethernet Advanced Protection Switching (EAPS) • SLA: 99.99% end-to-end throughput • SLA: 500 milliseconds maximum round-trip-time • SLA: 250 milliseconds maximum packet delay variation NZNOG 2006 – Victoria University Wellington, March 23 rd , 2006 6

University of Auckland 10 Gigabit campus ring (2)

Tamaki ITSS Epsom Symonds St NZNOG 2006 – Victoria University Wellington, March 23 rd , 2006 Newton Data Medicine 7

University of Auckland – test setup

Tamaki ITSS Epsom Symonds St NZNOG 2006 – Victoria University Wellington, March 23 rd , 2006 Newton Data Medicine 8

University of Auckland – throughput results

• 10 Gigabit backbone performance: full line rate for any packet sizes between 128 and 9212 bytes • Any single 1-Gigabit loop VLAN: loss free at any packet size • 10 loop 1-Gigabit VLAN performance (saturated core, with all services active): Packet size in bytes 64 128 256 512 1024 1280 1518 100% 8.12

4.71

2.54

1.31

0.66

0.51

0.42

Link utilization 99% 98% 97% 6.10

3.54

0.60

4.07

1.16

0.00

2.02

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

96% 0.00

0.00

0.00

0.00

0.00

0.00

0.00

NZNOG 2006 – Victoria University Wellington, March 23 rd , 2006 9

University of Auckland – delay and jitter results

• Fiber run length 44.7 km – 218.7 microseconds (RI=1.467) • OTD(min) = 258.4 microseconds • Switching, buffering accounts for 39.6 microseconds (15% of total) • Per switch latency 3.3 microseconds, inclusive serialization • RTTD(64) 517.0 microseconds at 100% of 1-Gigabit load • PDV(64) 1.2 microseconds  Meets SLA requirements comfortably, after adjustment for additional 10 km fiber spur NZNOG 2006 – Victoria University Wellington, March 23 rd , 2006 10

University of Auckland – latency as a factor of packet size

Packet size (in Minimum Latency (in nanoseconds) Average Maximum bytes) A -> B B -> A A -> B B -> A A -> B B-> A 64 258,436 258,385 258,549 258,483 259,465 258,591 128 261,985 261,934 262,047 262,001 262,089 262,088 256 265,226 265,175 265,296 265,239 265,381 265,278 512 273,662 273,611 273,717 273,699 273,817 273,766 1024 289,866 289,814 289,920 289,882 289,969 289,969 1280 299,537 299,485 299,581 299,565 299,691 299,640 1518 306,687 396,687 306,777 306,739 306,893 306,791 9212 579,630 579,629 579,727 579,689 579,784 579,835 Summary results (in nanoseconds) RTTD 64 PDV 64 A -> B -> A 516,821 A -> B -> A 517,032 1,235 A -> B -> A 518,056 NZNOG 2006 – Victoria University Wellington, March 23 rd , 2006 11

University of Auckland – EAPS protection switching

• Measured the impact of a ring break upon site-to-site traffic • Worst case scenario – re-routing once around the entire ring • Measurements conducted at single site • Metric – number of packets lost • Worst case timing was measured at 217 milliseconds • Typical timing is around 100-150 milliseconds  Exceeds SLA expectations for sub-second failover NZNOG 2006 – Victoria University Wellington, March 23 rd , 2006 12

University of Auckland – test summary

• Installation meets / exceeds original design parameters • Network is capable of delivering unconstraint full 1-Gigabit services across the metro domain • Fiber spur by far dominates latency (85%) • Switching latency is negligible, a few microseconds per packet at most • Serialization delay can become the dominating factor with large packet sizes and many hops  Consider the QoS impact when designing applications for either bulk data transfer or sensitive audio/video conferencing applications NZNOG 2006 – Victoria University Wellington, March 23 rd , 2006 13

REANNZ Advanced Network

Mount Albert 10Gb 10Gb Auckland 10Gb North Shore 10Gb North Shore 10Gb Rotorua

• 10 Gigabit backbone overlay on TelstraClear DWDM • Gigabit access in all major cities • Layer 2 network with QoS • Routers in Auckland, Wellington, Christchurch • International access: 155 Mbits/sec to Sydney, 622 Mbits/sec to California • Rollout in 2006 and 2007

Hamilton 10Gb Palmerston North 10Gb Palmerston North Wellington Nelson Napier 10Gb 10Gb Wellington 10Gb 10Gb 10Gb 10Gb 10Gb Christchurch 10Gb Lower Hutt Lincoln 10Gb 10Gb Havelock North 10Gb 10Gb

NZNOG 2006 – Victoria University Wellington, March 23 rd , 2006

= DWDM Connection = Fibre Connection = Fibre Patch Lead 10Gb Invermay

14

Dunedin

Acknowledgements and references

• University of Auckland ITSS department – Robert Beattie, Manager • REANNZ – Charles Jarvie, CEO • A cast of dozens of hands at TelstraClear and JazzTech

Thank you!

NZNOG 2006 – Victoria University Wellington, March 23 rd , 2006 15