Transcript Document

Adaptive Virtual Networking
For Virtual Machine-based
Distributed Computing
Peter A. Dinda
Prescience Lab
Department of Computer Science
Northwestern University
http://plab.cs.northwestern.edu
People and Acknowledgements
• Students
– Ashish Gupta, Ananth Sundararaj, Alex
Shoykhet, Jack Lange
• Collaborators
– In-Vigo project at University of Florida
• Renato Figueiredo, Jose Fortes
• Funders/Gifts
– NSF through several awards, VMWare
2
IBM xSeries
virtual cluster
(64 CPUs),
1 TB RAID
10/100
switch
Development cluster
(5 PowerEdge, 10 CPUs)
IBM zSeries mainframe
(1-way, 3.36TB storage)
Interactivity
Environment
Cluster, CAVE
(~90 CPUs),
8 TB RAID
2 Distributed
Optical Testbed
Clusters
IBM xSeries
(14-28 CPUs),
1 TB RAID
Nortel Optera
Metro Edge
Optical Router
Northwestern
Internet
GbE switch
IBM xSeries
Virtual cluster
(64 CPUs)
IBM xSeries
Dev. cluster
(8 CPUs)
RAID array
(1.2TB)
Sun Enterprise servers
(E450, E250; 6 CPUs)
Distributed Optical Testbed
(DOT) Private Optical Network
UFL
DOT clusters
with optical
connectivity
IBM xSeries
(14-28 CPUs),
1 TB RAID:
Argonne, U.Chicago,
IIT, NCSA, others 3
Users already know how to
deal with this complexity at
another level
4
Virtuoso:
“The Dell Model”
A. Shoykhet, J. Lange, and P. Dinda,
Virtuoso: A System For Virtual Machine
Marketplaces, Technical Report NWUCS-04-39, July, 2004.
R. Figueiredo, P. Dinda, J. Fortes, A
Case For Grid Computing on Virtual
Machines, Proceedings of the 23rd
International Conference on Distributed
Computing Systems (ICDCS 2003)
5
The Illusion
User’s
LAN
VM
User
Your machines are
sitting next to you.
6
Virtual Machines
• Language-oriented VMs
– Abstract interpreted machine, JIT Compiler, large library
– Examples: UCSD p-system, Java VM, .NET VM
• Application-oriented VMs
– Redirect library calls to appropriate place
– Examples: Entropia VM
• Virtual servers
– Kernel makes it appear that a group of processes are running on a separate
instance of the kernel or run OS at user-level on top of itself
– Examples: Ensim, Virtuozzo, UML, VServer, FreeVSD …
• Microkernels designed to host OSes
– Xeno VM
• Virtual machine monitors (VMMs)
– Raw machine is the abstraction
– VM represented by a single image
– Examples: IBM’s VM, VMWare, Virtual PC/Server, Plex/86, SIMICS,
Hypervisor, DesQView/TaskView. VM/386
7
Claim
• Virtual networking for VMs enables the
broad application of dream techniques…
– Adaptation
– Resource reservation
• … using existing, unmodified applications
and operating systems
– So actual people can use the techniques
8
Questions
• Is there enough application information?
– Resource demands
– Goals
• Is there enough resource information?
– Cycles
– Bandwidth
• Are there sufficient adaptation and
reservation mechanisms?
• Is the control loop fast enough?
9
Outline
• Motivation and claims
• VNET: A virtual network for virtual machines
– And what it enables
• VTTIF: Application topology inference
• Dynamic topology adaptation
– Combining VNET and VTTIF
• Current directions
• Conclusions
10
Why Virtual Networking?
• A machine is suddenly plugged into
your network. What happens?
– Does it get an IP address?
– Is it a routeable address?
– Does firewall let its traffic through?
– To any port?
How do we make virtual machine hostile
environments as friendly as the user’s LAN?
11
VNET: A Layer 2 Virtual Network
for the User’s Virtual Machines
• Why Layer 2?
– Protocol agnostic
– Mobility
– Simple to understand
– Ubiquity of Ethernet on end-systems
• What about scaling?
– Number of VMs limited (1024/user)
– Hierarchical routing possible because MAC
addresses can be assigned hierarchically
A. Sundararaj, P. Dinda, Towards Virtual Networks for
Virtual Machine Grid Computing, USENIX VM 2004
12
A Simple Layer 2 Virtual Network
Client
Server
SSH
VM monitor
Remote VM
Virtual
NIC
Physical
NIC
Friendly Local Network
Physical
NIC
Hostile Remote Network
13
A Simple Layer 2 Virtual Network
Client
Server
SSH
VM monitor
Remote VM
Virtual
NIC
Physical
NIC
Friendly Local Network
Physical
NIC
Hostile Remote Network
14
A Simple Layer 2 Virtual Network
Client
vnetd
UDP, TCP,
TCP/SSL, or
SSH tunnel
Server
vnetd
VM monitor
Remote VM
Virtual
NIC
Physical
NIC
Friendly Local Network
Physical
NIC
Hostile Remote Network
15
More Details
“eth0”
ethx
Client
LAN
Client
VNET
Proxy
Ethernet Packet
Captured by
Promiscuous
Packet Filter
ethz
ethy
IP Network
Ethernet Packet Tunneled
over TCP/SSL Connection
VM
“Host Only”
“eth0” Network
vmnet0
VNET
Host
Ethernet
Packet
Injected
Directly into
VM interface
A collection of such Proxy/Host connections forms a star
network centered at the Proxy on the user’s network
VNET 0.9 available from
http://virtuoso.cs.northwestern.edu
16
Initial Performance Results (LAN)
12
10
8
Faster than NAT approach
Lots of room for improvement
This version you can download
and use right now
6
4
2
0
17
VNET 1.0: Bootstrapping the Virtual Network
Host +
VNETd
Proxy +
VNETd
VM
• Star topology always possible
• Topology may change
• Links can be added or removed on demand
• Virtual machines can migrate
• Forwarding rules can change
• Forwarding rules can be added or removed on demand
18
Pseudo
proxy
Current Status Snapshots
19
VNET 1.0 Performance
• BW and latency similar to VNET 0.9
• Add/Delete Link: 21 ms
• Add/Delete Rule: 16 ms
• IBM e1350 cluster, 100 mbit switch
20
VNET 1.0 Topology Manipulation
(Eight VMs)
3.5
AlltoAll
Time in Seconds
3
Bus
Ring
2.5
Mesh
2
1.5
1
0.5
0
Setup
Teardown
21
VNET 1.0 Topology Manipulation
(Eight VMs)
5
AlltoAll - Bus
4.5
AlltoAll - Ring
Time in Seconds
4
AlltoAll - Mesh
3.5
Bus - Ring
3
Bus - Mesh
2.5
Ring - Mesh
2
1.5
1
0.5
0
Switch Forward
Switch Reverse
22
VNET 1.0 Topology Manipulation
(Eight VMs)
5
AlltoAll - Bus
4.5
AlltoAll - Ring
Time in Seconds
4
AlltoAll - Mesh
3.5
Bus - Ring
3
Bus - Mesh
2.5
Ring - Mesh
2
1.5
1
0.5
0
Switch Forward
Switch Reverse
23
Outline
• Motivation and claims
• VNET: A virtual network for virtual machines
– And what it enables
• VTTIF: Application topology inference
• Dynamic topology adaptation
– Combining VNET and VTTIF
• Current directions
• Conclusions
24
VM
Layer
Vnetd
Layer
Physical
Layer
25
Application communication
topology and traffic load;
application processor load
VM
Layer
Vnetd
Layer
Physical
Layer
26
Application communication
topology and traffic load;
application processor load
VM
Layer
Vnetd
Layer
Network bandwidth and
latency; sometimes
topology
Physical
Layer
27
Application communication
topology and traffic load;
application processor load
VM
Layer
Vnetd layer can collect all
this information as a side
effect of packet transfers
Vnetd
Layer
Network bandwidth and
latency, sometimes
topology; host load
Physical
Layer
28
Application communication
topology and traffic load;
application processor load
Vnetd layer can collect all
this information as a side
effect of packet transfers
and invisibly act
Network bandwidth and
latency; sometimes
topology
VM
Layer
Vnetd
Layer
Physical
Layer
29
Application communication
topology and traffic load;
application processor load
Vnetd layer can collect all
this information as a side
effect of packet transfers
and invisibly act
•VM Migration
Network bandwidth and
latency; sometimes
topology
VM
Layer
Vnetd
Layer
Physical
Layer
30
Application communication
topology and traffic load;
application processor load
Vnetd layer can collect all
this information as a side
effect of packet transfers
and invisibly act
•VM Migration
•Topology change
Network bandwidth and
latency; sometimes
topology
VM
Layer
Vnetd
Layer
Physical
Layer
31
Application communication
topology and traffic load;
application processor load
VM
Layer
Vnetd layer can collect all
this information as a side
effect of packet transfers
and invisibly act
•VM Migration
•Topology change
•Routing change
Vnetd
Layer
Network bandwidth and
latency; sometimes
topology
Physical
Layer
32
Application communication
topology and traffic load;
application processor load
Vnetd layer can collect all
this information as a side
effect of packet transfers
and invisibly act
•VM Migration
•Topology change
•Routing change
•Reservation
Network bandwidth and
latency; sometimes
topology
VM
Layer
Vnetd
Layer
Physical
Layer
33
Outline
• Motivation and claims
• VNET: A virtual network for virtual machines
– And what it enables
• VTTIF: Application topology inference
• Dynamic topology adaptation
– Combining VNET and VTTIF
• Current directions
• Conclusions
34
VTTIF: Application Traffic Load
Measurement and Topology Inference
• Parallel and distributed applications display
particular communication patterns on
particular topologies
– Intensity of communication can also vary from
node to node or time to time.
– Combined representation: Traffic Load Matrix
• VNET already sees every packet sent or
received by a VM
• Can we use this information to compute a
global traffic load matrix?
• Can we eliminate irrelevant communication
from matrix to get at application topology?
35
Traffic Monitoring and Reduction
Ethernet Packet Format:
ethz
VM
“Host Only”
“eth0” Network
vmnet0
SRC|DEST|TYPE|DATA (size)
VMTrafficMatrix[SRC][DEST]+=size
Each VM on the host contributes a row
and column to the VM traffic matrix
VNET
Host
Packets observed
here
Global reduction to find overall matrix,
broadcast back to VNETs
Each VNET daemon has a view of the
global network load
36
Denoising The Matrix
• Throw away irrelevant communication
– ARPs, DNS, ssh, etc.
• Find maximum entry, a
• Eliminate all entries below aa
• Very simple, but seems to work very
well for BSP parallel applications
• Remains to be seen how general it is
37
Offline Results: Synthetic Benchmark
38
NAS IS Benchmark
39
NAS IS Benchmark
h1
h1
h2
h3
h4
h5
h6
h7
h8
19.0
19.6
19.2
19.6
18.8
13.7
19.3
10.7
10.8
10.7
10.9
9.7
10.5
11.2
10.4
10.1
10.5
10.5
11.1
10.8
10.6
10.2
11.7
10.9
11.9
12.2
12.1
h2
22.6
h3
22.2
8.78
h4
22.4
8.9
9.5
h5
22.3
10.0
9.51
9.72
h6
24.0
8.9
10.7
9.9
10.8
h7
23.2
10.0
9.7
9.5
10.3
10.2
h8
24.9
11.2
11.0
11.8
11.5
11.2
12.0
10.7
*numbers indicate MB of data transferred.
40
Online Challenges
• When to start? When to stop?
– Traffic matrix may not be stationary!
• Synchronized monitoring
– All must start and stop together
41
When To Start? When to Stop?
Reactive Mechanisms
Start when traffic rate
exceeds threshold
Stop when traffic rate
exceeds a second threshold
Non-uniform discrete event
sampling
What is the Traffic Matrix
from the last time there
was at least one high
rate source?
Proactive Mechanisms
Provide support for queries
by external agent
Keep multiple copies of the
matrix, one for each
resolution (1s, 2s, 4s, etc)
What is the Traffic Matrix
for the last n seconds ?
42
Overheads (100 mbit LAN)
• Essentially zero latency impact
• 4.2 % throughput reduction versus VNET
A. Gupta, P. Dinda, Inferring the Topology and Traffic
Load of Parallel Programs Running In a Virtual Machine
Environment, JSSPP 2004.
43
Online: NAS IS on 4 VMs
44
Outline
• Motivation and claims
• VNET: A virtual network for virtual machines
– And what it enables
• VTTIF: Application topology inference
• Dynamic topology adaptation
– Combining VNET and VTTIF
• Current directions
• Conclusions
45
Dynamic Topology Adaptation
• VTTIF reactive mechanism run continuously
• On topology change, adjust VNET topology,
adding links in priority order
• Corresponding forwarding rules also added
• Measure performance (running time) of
application (BSP patterns application)
A. Sundararaj, A. Gupta, P. Dinda, Dynamic
Topology Adaptation in a Virtual Network of
Virtual Machines, In Submission
46
Inference and Adaptation (8 VMs, LAN)
90
Adapt
80
Infer
70
Seconds
60
50
40
30
20
10
0
All-to-All
Bus
Ring
Mesh
47
Example Result (all-to-all, 8 VMs, LAN)
1800
1600
Dynamic measurement and
reconfiguration
1200
1000
800
600
Full all-to-all network after
startup measurement
+ reconfiguration cost
Full all-to-all from
beginning of run
400
200
0
ideal
complete
star
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
Run Time (Seconds)
1400
No Fast Path Topology
Number of Fast Path Links in Virtual Topology
48
Outline
• Motivation and claims
• VNET: A virtual network for virtual machines
– And what it enables
• VTTIF: Application topology inference
• Dynamic topology adaptation
– Combining VNET and VTTIF
• Current directions
• Conclusions
49
Adaptation With Migration
• Learn how to adapt using Virtuoso’s VM
migration capabilities
• Virtuoso migration times with rsync
– ~300 seconds (1.1 GB machine)
– ~50 seconds (100 MB machine)
• Versioning file system approaches
• Data point: CMU ISR project: 2.5-30 seconds
for personal windows VM
50
Policy Avoidance Routing
• Multi-site collaborations often stymied by
interactions between per-site network security
policies
• VNET opportunity: find a path on behalf of
application where one exists, but is obscured
• Example: NAT Traversal
– RFC 3489 / STUN (chownat)
• Example: Tunneling through initiation protocol
– HTTP or SSH
51
Physical Network Measurement
• Use existing application traffic to
measure underlying physical network
• Passive packet dispersion techniques
– With Bruce Lowekamp, W&M
• Topology inference
– With Bruce Lowekamp, W&M
M. Zangrilli and B. Lowekamp, Using Passive Traces
of Application Traffic in a Network Monitoring System,
HPDC 2004.
52
Integration
With Resource
Prediction
Visit rps.cs.northwestern.edu for more info and downloads
53
Improving the Backbone
• Replacing the proxy star with a
multisource muliticast system for higher
performance and resilience
• FatNemo protocol
– Arrange nodes into fat tree
S. Birrer, D. Lu, F. Bustamante, Y. Qiao, P. Dinda,
FatNemo: Building a Resilient Multi-Source Multicast
Fat-Tree, WCCD 2004
54
Extended Application Inference
• Offered computational load
• VM-internal performance data
• Synchronization points and waiting
• Inference of application goals
• Simple layered API for getting more
application information into system
55
Making the Fast Path Fast
• Move VNET forwarder into kernel of host OS
• Guest OS device driver to directly
communicate out of VM to VNET Forwarder
• Inference may make deposit message
passing possible
• Goal: Minimal overhead BW and latency for
using VNET, even on gigabit and faster
networks
56
Leveraging Optical Networking
• Use inferred application topology to do
light path setup on behalf of application
• Currently: ICAIR ODIN system, DOT
network
57
Related Work
•
Collective / Capsule Computing (Stanford)
– VMM, Migration/caching, Hierarchical image files, Attestation
•
Internet Suspend/Resume (CMU/Intel)
– Your VM follows you around (will be deployed on CMU campus)
•
Denali (U. Washington)
– Highly scalable VMMs (1000s of VMMs per node)
•
•
•
CoVirt (U. Michigan)
Xenoserver (Cambridge)
SODA (Purdue)
– Virtual Server, fast deployment of services
•
Ensim
– Virtual Server, widely used for web site hosting
– WFQ-based resource control released into open-source Linux kernel
•
Virtouzzo (SWSoft)
– Ensim competitor
•
Available VMMs: IBM’s VM, VMWare, Virtual PC/Server, Plex/86,
SIMICS, Hypervisor, DesQView/TaskView. VM/386
58
Conclusions
• Virtual machines on virtual networks as
the abstraction for distributed computing
• Virtual network as a fundamental layer for
measurement and adaptation
• Status
– Virtuoso prototype running on our cluster
– VNET 0.9 released.
– VNET 1.0 (with VTTIF) in progress
– Wayback versioning file system released
59
For More
Information
• Prescience Lab
– http://plab.cs.northwestern.edu
• Virtuoso
– http://virtuoso.cs.northwestern.edu
• Join our user comfort study!
– http://comfort.cs.northwestern.edu
• Join our intrusion detection study!
– http://ga-ids.cs.northwestern.edu
60