Transcript Recommending a Strategy - National University of Singapore

The Network
Affandi
Singaren, Singapore
Doug Pearson
Indiana University
Internet2 Commons Site Coordinator Training
December 3, 2003
National University of Singapore
A Sample University
Network
 LAN
 Building network
 Campus backbone
 WAN intranet
 Internet
 Internet2
2
Sample Network
3
Wiring and Station
Cables
Ethernet switch
horizontal wiring
datajack
station cable
device
iMac
3rd floor
Common practice is unshielded twisted-pair
(UTP) according to the specs:
• CAT 3 [old] supports 10 Mbps Ethernet (10base-T)
• CAT 5 [modern] supports 10base-T, 100 Mbps
(100base-TX) and 1000 Mbps (1000base-T) Ethernet
• CAT 6 [new] supports CAT 5 applications +
4
Wiring and Station
Cables
Actual wire used and quality of
installation may vary widely – know
your wiring!
Important to consider the station cables
• Don’t use sub-CAT 5 station cables for 100 Mbps
connections.
5
Ethernet LAN
Ethernet switch
horizontal wiring
datajack
station cable
device
iMac
3rd floor
2nd floor
1st floor
router
6
Ethernet LAN
10 / 100 / 1000 Mbps
Full- and Half-Duplex
• Half-duplex: send or receive, one at a time.
• Full-duplex: send and receive simultaneously.
• 10 Mbps Ethernet supports half-duplex; full-duplex
is not consistently implemented.
• 100 Mbps supports half- and full-duplex.
7
LAN: Switches vs.
Repeaters
 Repeaters (hubs) are old technology.
 A repeater sends (repeats) packets
that are incoming on one port, out all
other ports (I know you’re out there
somewhere!).
 Can only operate in half-duplex mode.
 Bandwidth and jitter provided to any
single device is highly dependent on
the LAN traffic.
8
LAN: Switches vs.
Repeaters
 A switch learns the MAC addresses of the
devices connected to it, and sends packets
directly and only to the target end-point.
 Provides much more consistent bandwidth
and latency (low jitter).
 A well-designed switched LAN is important
for videoconferencing. Repeater-based
LANs should be upgraded to switched for
videoconferencing!
9
LAN: Ethernet Duplex
Mismatch
 “One of the most common causes of
performance issues on 10/100Mb
Ethernet links is when one port on the
link is operating at half-duplex while
the other port is operating at fullduplex.”
• http://www.cisco.com/warp/public/473/3.html
10
LAN: Ethernet Duplex
Mismatch
“There is a silent performance-killer out
there, one so inconspicuous that it is hardly
ever looked for or even suspected. You
could suffer from it and never know it, as it
robs a site of performance but not
connectivity. This performance-killer has a
name: Ethernet duplex mismatch.”
• http://www.hostingtech.com/nm/01_01_mismatch.html
11
LAN: Ethernet Duplex
Mismatch
If one end of a connection (device or
Ethernet switch) is set for autonegotiation, and fails to see autonegotiation at the other end, the former
sets itself to the default, half-duplex.
Auto-negotiation can sometimes fail,
even when both sides are set to auto
(although this isn’t as prevalent as in
the past).
12
LAN: Duplex Mismatch
– Detection
Microsoft Windows doesn’t display
the auto-negotiated duplex setting.
Some routers re-negotiate Autoduplex, which introduces jitter.
13
LAN: Ethernet Duplex
Mismatch
SETTINGS
switch
device
auto
half
full
auto
half
full
auto
full
full
auto
auto
half
half
auto
RESULTS
switch
device
BAD!
BAD!
auto
half
full
auto
half
full
half
full
full
half
half
half
half
half
14
LAN: Duplex Mismatch
– Detection
 Show port statistics on the Ethernet switch. When
mismatched, the full-duplex end will report a high
level of CRC or alignment errors; the half-duplex
end will report a high number of late collisions.



PortAlign
FCS Xmit
Rcv UnderSize
Err Err Err Err
2/11-
0
0
3077
0


Port Single-
Multi- Late-

Coll
Coll

2/11 3233
Coll
0
2588
0
Excess- CarriSen
Runts
Giants
Coll
0
2489
0
15
LAN: Duplex MismatchPrevention
Always configure switches and devices
according to your local policy. An
example policy is:
• If building wiring is sub-CAT 5, then set switch
ports to 10/half
• If building wiring is CAT 5 or better, then set switch
ports and devices to Auto.
Monitor switch port stats and logs
16
Router
Ethernet switch
3rd floor
2nd floor
1st floor
router
Segments LANs into
distinct networks and
subnetworks, e.g., the
distinct red, green, and
blue LANs with distinct
network numbers.
Segments LANs into
broadcast domains
17
Router
Provides interface to
the WAN.
• Intranet, commercial
Internet, and Internet2
connections.
• Typically, every
networked device at an
Internet2-connected
institution has
connectivity to Internet2.
18
VLAN
A single, physical LAN can be logically
segmented into multiple logical LANs;
and,
Physically separate LANs can be made
to behave and appear as a single LAN.
19
VLAN
Packets are tagged according to LAN
membership, e.g., green LAN, red
LAN, and blue LAN.
Ethernet switches establish broadcast
domains according to the defined
VLAN boundaries.
Routers establish multiple VLANs on a
single interface.
20
VLAN
21
VLAN
Modern campus network architectures
are tending to move away from
traditional router-for-a-building design,
to VLAN designs.
22
Old Design Included a
lot of Routers
routers
23
New Design Includes
VLAN Router
router
24
WAN Segments
Service
Speed
Sample Uses
T1
DS3
1.5 Mbps
45 Mbps
remote building; extension center
inter-campus; Internet (I1) connection
OC3
155 Mbps
inter-campus; I1 & Internet2 connection
OC12
622 Mbps
I1 backbones; Internet2 connection
OC48
2.4 Gbps
Gigabit
Ether
net
1 Gbps
I1 and Internet2 backbones
advanced inter-campus connections
when have access to dark fiber
25
Indiana University Abilene
NOC Weathermap

26
High Performance Research and
Education Networks
Internet2 / Abilene
• http://www.internet2.edu
• http://www.abilene.iu.edu
STARTAP and International Networks
• http://www.startap.net
US Government-Sponsored Networks
• http://www.startap.net/NETWORKS
27
Traffic on the Network
Typical university today:
• IP
– TCP
– UDP
• IPX [diminishing]
• Appletalk [diminishing]
28
Traffic on the LAN
Unicast : one-to-one
Multicast: one-to-many
Broadcast:
one-to-every
29
Unicast
Most common traffic
Common applications: mail,
Web browsing, file transfer, etc.
30
IP Multicast
A one-to-many mode of transmission
Network numbers 224.0.0.0 through
239.255.255.255 are reserved for
multicast.
Examples of multicast applications:
• Vic/rat videoconferencing
• Centralized PC software administration tools such
as Symantec Ghost
31
IP Multicast – Leak
Problems
Beware: high rates of unpruned
multicast can adversely affect
videoconference performance.
Use a network traffic and protocol
analyzer to identify this problem.
32
Broadcast
 A one-to-every mode of transmission
 Used by network protocols including ARP
and IPX, NetBIOS system discovery, and
name resolution.
 All devices on the network must process
every broadcast packet; high broadcast rates
can divert processing capacity.
 If the broadcast domain is too large or
unusually active, the activity required at the
end-point to deal with the broadcasts could
diminish performance.
33
Broadcast
A healthy network should have less
than 100 broadcast packets per
second.
Check using a network traffic and
protocol analyzer tool.
34
Firewalls
 A firewall is a network node that acts to
enforce an access control policy between
two networks, e.g., between a university
intranet and the commercial Internet.
 Used to secure IT resources against external
attacks and break-ins.
 Network-layer firewalls typically make their
decisions based upon port numbers and
source/destination addresses.
 Application-layer firewalls act as proxies.
35
Firewalls
H.323 uses the IP ports:
• Statically-assigned TCP ports 1718 – 1720 and
1731 for call setup and control.
• Dynamically-assigned UDP ports in the range of
1024 – 65535 for video and audio data streams.
Firewalls don’t allow unrestricted ports.
Typical modern firewalls and H.323
don’t get along so well.
36
Firewalls – Solutions for
H.323
 [bad; non-scaleable] Allow unrestricted
ports for specific, known, external IPaddresses.
 [better, but still not so good] Use feature of
some videoconferencing clients to confine
dynamic ports to a specific, narrow range.
 [OK, but extra admin work and cost] Use an
H.323 application proxy.
 [best] Use a firewall that snoops on the
H.323 call set-up channels (static ports) and
opens ports for the audio/video (dynamic
ports) as needed.
37
NATs
 Allows multiple computers behind the NAT to
share one external network address.
 Uses:
• Alleviate shortage of IP addresses
• Security – obscures view of the network from
outside
• Flexible network administration
 Not commonly used at universities on the
campus level. Used somewhat in
corporations. Common in small offices and at
home – behind DSL, cable modem, or ISDN
network service.
38
NATs
 Difficult to use H.323 behind NATs.
 Some videoconferencing terminals provide
features to work with NAT – refer to
videoconferencing terminal documentation.
39
Latency
 Latency is the time required for a packet to
traverse a network from source to
destination.
 Components of latency include:
• Propagation delay: the time it takes to traverse
the distance of the transmission line; controlled by
the speed of light in the media; rule-of-thumb:
20ms San Francisco to New York.
40
Latency
 Transmission delay: the time it takes for the
source to put a packet on the network. Ruleof-thumb: < 1ms.
 Store-and-forward delay: the cumulative
length of time it takes the internetworking
devices along the path to receive, process,
and resend the packets. Rule-of-thumb:
variable, and depends upon network load.
41
Latency
 Rule of thumb:
• A one-way delay of:
• 0 – 150 ms provides excellent interactivity
• 150 – 300 ms is OK
• 300 – 400 ms is bad
• 400+ ms is unacceptable
42
Jitter
Jitter is variation in latency over time.
If the endpoints are on switched LANs,
then the primary source of jitter is
variation in the store-and-forward time,
resulting from network load.
H.323, particularly audio, is adversely
affected by high levels of jitter.
What is high? Rule of thumb?
43
Packet Loss
Packet loss is typically due to
congested links and routers.
• 1% is noticeable
• 5% becomes intolerable
44
QoS
Not currently feasible on commercial
Internet and Internet2 networks for
production, regular use. Internet2 is
working on QoS plans, but the current
over-provisioned Internet2 network
doesn’t dictate need.
Is useful on over-utilized intranet WAN
links.
45
QoS
How:
• Some videoconferencing terminals can set
the IP precedence bits. Use that for
marking and priority queuing on the WAN.
 Or:
• Use a H.323 Proxy for consolidation of
traffic to a single address, router access list
for marking, and priority queuing on the
WAN.
46
QoS
Caution!
• The wrong implementation could result in
unwanted tradeoffs, e.g., packet loss
improves but jitter gets worse.
47
The End-to-End
Performance Problem
Scenario
• Users on two different campuses of a university
are experiencing poor video and audio in a
conference.
• Each user is supported by a different group of
videoconferencing engineers.
• Each campus is supported by a different group of
network engineers.
• The wide-area network is supported by a third
group of network engineers.
48
The End-to-End
Performance Problem
Problem
• How do the users get timely, useful assistance?
• How is network problem resolution coordinated?
49
The End-to-End
Performance Problem
Obstacles
• Different groups, schedules, and priorities.
• No one engineer has a complete understanding of
the entire network path.
• No one engineer can gain access to all the
network nodes (routers, switches) along the path
to inspect for trouble.
• Communications are inconsistent from engineer to
engineer.
50
The End-to-End
Performance Problem
Solutions
• Articulate the E-2-E problem to network
management and engineers on all campuses.
• Establish reliable communication tools, and insist
that engineers utilize the tools.
• Hold regular meetings; bring all engineers
together in one place and time to share
information.
• Have good network documentation for all
networks.
51
H.323 is Network
Sensitive!
The big problems are:
• Half/Full-duplex mismatches
• Packet loss
• Jitter
• Substandard horizontal wiring or station cables
• Multicast leaks
• High broadcast rates
52
Tools










Ping – availability, loss, roundtrip time
Traceroute – path discovery
Pingplot – graphical traceroute/ping
MRTG – graph link/port utilization & errors
Iperf – bandwidth, loss and jitter
Gnuplotping – visualize jitter
Sniffer – inspect traffic on the LAN
VideNet Scout – bandwidth, loss and jitter
Internet2 Detective – detect I2 connection
H.323. Beacon –protocol-specific tests
53
Tools: Ping
Test for availability, loss, and roundtrip
time
ICMP Echo Request
• Plus optional dummy payload – only in the
direction of the ping, i.e., source  destination
54
Tools: Sample Ping from
Windows
 C:\WINDOWS>ping 10.1.1.1
 Pinging 10.1.1.1 with 32 bytes of data:
 Reply from 10.1.1.1: bytes=32 time=88ms TTL=112
 Reply from 10.1.1.1: bytes=32 time=72ms TTL=112
 Reply from 10.1.1.1: bytes=32 time=69ms TTL=112
 Reply from 10.1.1.1: bytes=32 time=69ms TTL=112
 Ping statistics for 10.1.1.1:

Packets: Sent = 4, Received = 4, Lost = 0 (0%
loss),
 Approximate round trip times in milli-seconds:

Minimum = 69ms, Maximum = 88ms, Average =
74ms
 C:\WINDOWS>
55
Tools: Sample Ping from
Windows

C:\WINDOWS>ping -l 40000 10.1.1.1

Pinging 10.1.1.1 with 40000 bytes of data:

Reply from 10.1.1.1: bytes=40000 time=2412ms TTL=112

Reply from 10.1.1.1: bytes=40000 time=2721ms TTL=112

Reply from 10.1.1.1: bytes=40000 time=2761ms TTL=112

Reply from 10.1.1.1: bytes=40000 time=2714ms TTL=112

Ping statistics for 10.1.1.1:

Packets: Sent = 4, Received = 4, Lost = 0 (0%
loss),

Approximate round trip times in milli-seconds:

Minimum = 2412ms, Maximum =
2652ms

C:\WINDOWS>
2761ms, Average =
56
Tools: Ping Big and Fast
on Linux
 ping -c2000 -i.03 -s1470 -q
[destination]
• Count of 2000 packets
• Interval of .03 seconds between packet starts
• Packet size of 1470 bytes
• (2000)(.03) = 60 second long test
• (1/(.03 sec/packet))(1470 bytes/packet)(8
bits/byte) = 392 Kbps
57
Tools: Ping Big and Fast
on Linux
 $ ping -c2000 -i.03 -s1470
[hostname]
-q
 PING [hostname] ([hostaddr]) from
[hostaddr2] : 1470(1498) bytes of data.
 --- [hostname] ping statistics -- 2000 packets transmitted, 2000 packets
received, 0% packet loss
 round-trip min/avg/max = 4.8/5.1/13.2
ms
58
Tools: Traceroute
 Used to discover the layer-3 network path
(routers) between the two endpoints
 Doesn’t identify layer-2 devices (switches)
 Must run from one of the discovery
endpoints – it can’t act as a third party.
 Take baselines – know what your path
should be in advance of trouble
59
Tools: Traceroute
Microsoft tracert uses ICMP packets,
whereas Unix traceroute uses UDP;
may be of importance in networks
where routers are configured to not
respond to ICMP; or if ICMP is blocked.
60
Tools: Traceroute;
Sample Output
 [dodpears@huck dodpears]$ traceroute www.internet2.edu
 traceroute to www.internet2.edu (209.211.239.208), 30 hops max, 38 byte
packets
 1 wcc-sub5-hp1 (129.79.5.253) 11.726 ms 0.627 ms 0.571 ms
 2 iub-gw (129.79.8.10) 3.133 ms 0.717 ms 0.651 ms
 3 156.56.249.22 (156.56.249.22) 2.544 ms 3.138 ms 2.538 ms
 4 abilene-iupui.abilene.ucaid.edu (198.32.11.13) 5.245 ms 3.402 ms
3.493 ms
 5 clev-ipls.abilene.ucaid.edu (198.32.8.26) 9.381 ms 9.586 ms 9.244 ms
 6 nycm-clev.abilene.ucaid.edu (198.32.8.30) 23.198 ms 21.963 ms
21.775 ms
 7 border-abilene-oc3.advanced.org (209.211.237.97) 23.448 ms 23.268
ms 23.052 ms
61
 8 www.internet2.edu (209.211.239.208) 23.559 ms 23.478 ms 23.234 ms
Tools: Traceroute
 8 www.internet2.edu (209.211.239.208) 23.559 ms 23.478 ms 23.234
ms
 Hop
 Router/host name


Router/host address
Round-trip times of each of three probes
62
Tools: Traceroute –
Example Uses
Identify the path, and then perform
pings along the path segments to
isolate troublesome segments.
Insure that Internet2 is being used for a
connection, rather than commercial
Internet.
63
Tools: Ping Plotter
Shareware tool; $15
• http://www.pingplotter.com
Performs a visual traceroute and ping
tests along the entire path
Permits identification of bottlenecks
along a path
64
Tools: Ping Plotter
65
Tools: MRTG
Multi Router Traffic Grapher
Collect and graph scalar, time-based
data, e.g., router and link performance
data.
66
Tools: MRTG
67
Tools: MRTG
68
Tools: Iperf
http://dast.nlanr.net/Projects/Iperf/
Client/server application that
• Measures maximum TCP bandwidth
• Facilitates tuning of TCP and UDP parameters
• Reports bandwidth, jitter, and packet loss
69
Tools: Iperf
Example on Intercampus DS3
At server, invoke:
iperf -fk -i30 -u -s
(f)ormat reports in kbps
(i)nterval for reporting = 30 seconds
(u)dp
(s)erver mode
70
Tools: Iperf
Example on Intercampus DS3
At client, invoke:
iperf -u -b800k -t3600 -c [hostnameserver]
(u)dp
(b)andwidth = 800kbps
(t)ime of run = 3600 seconds
(c)lient mode
[hostname-server] = server to target
71
Tools: Iperf
Example on Intercampus DS3
 [dodpears@vc-iperf iperf]$ iperf -fk -i30 -u -s
 ----------------------------------------------------------- Server listening on UDP port 5001
 Receiving 1470 byte datagrams
 UDP buffer size: 64.0 KByte (default)
 ----------------------------------------------------------- [
3] local 149.166.197.80 port 5001 connected with 129.79.92.230 port 1031
 [ ID] Interval
Datagrams
Transfer
 [
3]
0.0-30.0 sec
3000 KBytes
819 Kbits/sec
0.300 ms
0/ 2090 (0%)
 [
3] 30.0-60.0 sec
3000 KBytes
819 Kbits/sec
0.242 ms
0/ 2090 (0%)
 [
3] 60.0-90.0 sec
3000 KBytes
819 Kbits/sec
0.338 ms
0/ 2090 (0%)
9000 KBytes
819 Kbits/sec
0.263 ms
72
0/ 6393 (0%)
Bandwidth
Jitter
Lost/Total
 [...]
 [
3]
0.0-90.0 sec
Iperf- Example on
Intercampus DS3
 MRTG utilization graph
showed bandwidth peaking at
capacity ~ 10:00a – 2:00p
 As utilization peaked on the
DS3, jitter measured by Iperf
rose to unacceptable level
 Iperf also reported periodic
high packet loss, with no
apparent correlation to the
low-resolution MRTG
utilization reports
73
Iperf- Example on
Intercampus DS3
 Second day, utilization as
reported by MRTG is
staying reasonable.
 Jitter measured by Iperf is
staying low.
 The periodic high packet
loss remains, until noon
when network engineer
adjusted the QoS settings.
74
Iperf- Example on Intercampus
DS3
 Third day, utilization
peaking at 3:00p.
 As utilization peaked
jitter measured by Iperf
also rising.
 Still no packet loss.
 QoS fixed the packet loss
problem, but still not
certain about the jitter –
more analysis needed.
75
Tools: Gnuplotping
Pings multiple hosts in parallel with
graphical display (gnuplot) of the delay
distribution.
Runs on Unix/X-Windows
76
Tools: gnuplotping
77
Tools: Network Traffic
Analyzer
Reveals the traffic on a LAN
Protocol analysis
Reports such as utilization, protocols,
conversations, nodes, etc.
Network General Sniffer
WildPackets EtherPeek
78
Tools: ViDeNet Scout
Scout is a web-based, distributed
network performance analysis tool
developed at the University of North
Carolina at Chapel Hill.
Scout makes use of the Chariot
performance testing engine developed
by NetIQ.
• http://scout.video.unc.edu/
79
Tools: Internet2
Detective
I2 Detective is a small application.
Detects Internet2 connection.
Measures connection bandwidth (using
Iperf).
Detects multicast connection.
• http://detective.internet2.edu/
80
Tools: H.323 Beacon
 Used to measure, monitor and qualify the
performance of an H.323 Videoconference
session.
 Provides H.323-protocol specific evidence
and other information necessary to
troubleshoot H.323 application performance
problems in the network and at the host
(end-to-end)
• http://www.itecohio.org/beacon/
81
Recommendations
 Develop a close relationship with the
network engineers and NOC. Make sure
they understand what’s being done with
videoconferencing and the network
sensitivity of IP-based video.
 Articulate the End-to-End Performance
Problem to network engineering and
operations management. Champion ways to
reduce the problem.
 Be sure to open trouble tickets with your
NOC so that a problem history is
maintained.
82
Recommendations
Use switched Ethernet.
Watch out for duplex mismatches.
Keep an eye on utilization of WAN
links, packet loss, and jitter.
Make sure you don’t have broadcast or
multicast leaking problems.
Make sure wiring is up to the task.
83
Recommendations
Have engineers in the
videoconferencing support group
trained to understand networking
issues and tools.
84