Transcript VTL: A Transparent Network Service Framework

Symbiotic Virtualization
John R. Lange
Thesis Proposal
Department of Electrical Engineering and Computer Science
Northwestern University
June 2009
Introduction
• VMs are traditionally Black boxes
– Separated from the VMM by a semantic gap
– Does provide a clean interface
• Does that make sense in today’s environment?
– Cloud computing, live migration, differing architectures
– Guests should know they are in a VM
• Many reasons to bridge the gap
– Performance, Security, Monitoring, etc…
• Existing approaches don’t allow this
• Symbiotic Virtualization is an alternative to black box design
2
Symbiotic Virtualization
• Novel approach to designing VMMs and operating
systems
• OS compatible with native hardware interface
• BUT also optionally exposes a software interface that
can be used by a VMM
• Essentially, the VMM can easily inspect and modify
the guest OS
– Optional and Incremental
3
Outline
•
•
•
•
•
•
The Semantic Gap
Thesis Statement
Palacios and Kitten
Symbiotic Virtualization
Schedule
Contributions
4
Semantic Gap
• VMM architectures are designed as black boxes
– Explicit OS interface (hardware or paravirtual)
– Internal OS state is not exposed to the VMM
• Many uses for internal state
– Performance, security, etc...
– VMM must recreate that state
• “Bridging the Semantic Gap”
• Many examples
– Virtuoso Project
– Lycosid, Antfarm, Geiger, IBMon, many others
5
Virtuoso Project
•
Bridged the semantic gap for virtual networking
–
Examine physical network traffic to model application behavior
•
Provide virtual services to unmodified OSes and Applications
•
Virtuoso Project Components
–
VNET
•
–
VTTIF
•
–
Sundararaj, A., Gupta, A., and Dinda, P. Increasing application performance in virtual environments through
run-time inference and adaptation. In Proceedings of the 14th IEEE International Symposium on High
Performance Distributed Computing
VRESERVE
•
–
Gupta, A., and Dinda, P. Inferring the topology and traffic load of parallel programs running in a virtual
machine environment. In Proceedings of the 10th Workshop on Job Scheduling Strategies for Parallel Processing
VADAPT
•
–
Sundararaj, A., and Dinda, P. Towards virtual networks for virtual machine grid computing. In Proceedings of
the 3rd USENIX Virtual Machine Research And Technology Symposium (VM 2004)
Lange, J., Sundararaj, A., and Dinda, P. A. Automatic dynamic run-time optical network reservations. In
Proceedings of the 14th IEEE International Symposium on High Performance Distributed Computing
VTL
•
Lange, J. and Dinda, P. Transparent network services via a virtual traffic layer for virtual machines. In
Proceedings of the 16th International Symposium on High Performance Distributed Computing
6
VNET
• Overlay network for virtual machines
– Remotely distributed VMs appear connected to a LAN
– Layer 2 overlay, operates on ethernet frames
• Supports arbitrary overlay topologies, routing, and
link types
• Provides mechanisms to maximize network
performance
7
VTTIF and VADAPT
• Virtual Topology and Traffic Inference
Framework
– Infers communication topology and traffic load
matrix for a VM
• VADAPT
– Uses information from VTTIF
– Adaptively optimizes VNET overlay topology
8
VRESERVE
• Automatic and dynamic network reservations
– Allows unmodified applications to use circuit
switched optical networks
• Added optical network reservation interface to
VNET
– Automatically reserves network link when VTTIF
detects traffic between two connected hosts
9
VTL: Transparent Network Services
• Manipulate data and signaling of connections to add
services to existing unmodified applications and OSes
– High Level transformations of Low Level traffic
– Transparency: Manipulations invisible to guest
environment (Black Box approach)
• VTL (Virtual Traffic Layer)
– A framework for creating Transparent Network Services
• Can transform TCP connections into different
protocols
10
Bridging the Semantic Gap
• Enables many useful features and
optimizations
• However…
– Current approaches are labor intensive
• Reverse engineering an OS
– Highly specific to OS implementation
– Collected information not always accurate
11
Symbiotic Virtualization
• Bridging the semantic gap is hard
– Can we design a virtual environment with no gap?
• Symbiotic Virtualization
– Design both guest OS and VMM to minimize semantic gap
– 2 components
• Guest OS provides internal state to VMM
• Guest OS services requests from VMM
– Interfaces are optional
• Not required for correct operation
12
Thesis Statement
• I propose symbiotic virtualization, an approach to OS design that preserves the
benefits of full system virtualization, while enabling performance and
functionality benefits.
• In symbiotic virtualization, an OS targets the native hardware interface and can
run unmodified on raw hardware. However, it also exposes a software interface
that can be leveraged by a symbiotic virtualization-aware VMM.
• Both the interface and its use by the VMM are optional, but if it exists, and the
VMM uses it, the VMM and the OS can mutually benefit.
• Symbiotic virtualization is markedly different from the current virtualization
approaches, and is best considered as being on a continuum between full system
virtualization and paravirtualization.
13
Thesis Goals
•
•
•
•
•
Define and formalize Symbiotic Virtualization
Develop formal symbiotic interfaces
Implement symbiotic interfaces inside an OS
Implement set of symbiotic extensions
Use examples to evaluate the symbiotic
approach
14
Palacios
• OS independent embeddable VMM
– Written from scratch at NU and UNM
• Designed to be modularly linked into existing kernels
– Minimal host OS interface
– Compiles into static library
– Currently embedded: Kitten and GeekOS
• Open Source (BSD License)
– Downloaded ~1000 times
• Lead developer
15
Palacios Details
• Supports 32 and 64 bit environments
– Host and Guest
• Full hardware virtualization
– Currently only supports AMD extensions
– Intel VMX in process
• Supports Linux and HPC guest OSes
• Relatively small: ~28K lines
16
Architecture
Palacios
17
Kitten
• Lightweight HPC OS from Sandia National Labs
– Designed for large scale HPC systems (Cray XT)
– Successor to Catamount and earlier lightweight kernels
• Based on Linux
– Only the necessary components
– Limited Linux ABI compatibility
• Uses Palacios for virtualization
– Embedded as a library
– VMs launch as part of job submission
• Contributing developer
18
Palacios as an HPC VMM
•
Minimalist interface:
– Does not require extensive host OS features
– Easily embedded into even small kernels
•
Full system virtualization:
– Does not require guest OS changes
– Runs existing kernels without any porting
• Kitten, Catamount, Cray CNL, and IBM’s CNK
•
Contiguous memory preallocation:
– Preallocates guest memory as a physically contiguous region
– Vastly simplifies the virtualized memory implementation
– Deterministic performance for most memory operations
•
Passthrough resources and resource partitioning:
– Host resources are easily mapped directly into a guest environment
– Provides access to high performance devices, with existing device drivers, with no
virtualization overhead.
•
Low noise:
– Minimizes the amount of OS noise injected by the VMM layer.
– No internal timers and no accumulated deferred work.
19
Symbiotic Virtualization in HPC
• HPC environments are well suited to symbiotic techniques
• Full trust of the software stack
– Fewer security concerns
• Specific hardware configurations
– Limited number of devices
• Constrained problem space
– Small number of applications
• Implementations can be very specific
• Environments are much smaller
– Internal OS state is simpler than a general purpose OS
• At large scale performance impact is dramatic
– Large impetus to optimize VMM and OS
20
HPC Performance Example
• Guest OS behavior can differ widely
– Must optimize for specific OSes and applications
• Example:
– Catamount and Compute Node Linux
• 2 HPC OSes
– Process switching implementation
• CNL swaps page tables
• Catamount does not
– Nested and shadow page tables have very different performance
characteristics
– Evaluated with 2 HPC benchmarks
• HPCCG and CTH
• 3 configurations (Native, Shadow Paging, Nested Paging)
• Running on RedStorm Development Cages (Cray XT)
21
HPCCG Benchmark
Compute Node Linux
Catamount
22
CTH Benchmark
Compute Node Linux
Catamount
23
Takeaway
• At large scale minor performance problems become large
– Very important to minimize any performance overhead introduced
• VMM needs to know about guest internals
– Should modify behavior for each guest environment
– Which paging method to use depends on guest
• Inference is not desirable in HPC environment
– Unacceptable performance overhead
– Convergence time
– Mistakes have large consequences
• Symbiotic approach is very appealing
24
Symbiotic Virtualization
• Definition based on formalization
– Formalized interfaces
• Two types of interfaces
– Passive information interface
• VMM can read guest OS state
– Functional interface
• VMM can send requests to guest OS
• Neither required for OS to function correctly
– Symbiotic OS can run on hardware
– Non-symbiotic OS can run on symbiotic VMM
– Can be implemented incrementally
25
Passive Interface
• Formalize the interface for bridging the semantic gap
– Ideally removes the gap
• Internal state already exists but it is hidden
– Existing tools try to recreate this data in the VMM
• Symbiotic Interface:
– Structure internal OS state in a way that is easily parsed
• Semantically rich
– Expose OS state to the VMM
• Easily accessible
26
Example interface
• Linux process list
– Organized in a series of lists
– Scattered throughout kernel address space
– Lots of information included inside
• Priority, memory map, open file descriptors, etc
• Symbiotic Interface
– Collect task information in standard location
– Organize information to be easily parsable
• Reserved memory page that holds pointers to high priority
processes
• List of CR3 values that should be cached
27
Functional Interface
• Mechanism for OS to expose functionality to
VMM
– Guest OS services VMM requests
• Possible interfaces
– Guest OS notifications
– VMM can force explicit upcalls
– Iterator based system
28
Initial Functional Interface
• Partial initial test implementation
– Prosnitz and Xia
• Implemented inside GeekOS and Palacios
• Iterator based
– Modelled on RPCs
29
Issues
• New VMM/OS interaction model
• Traditional virtualization assumptions no
longer true
– No longer a black box
• Some new issues to be addressed
– Trust
– Design Complexity
30
Symbiotic Trust Model
• Current Architectures: unidirectional trust
– Guest OS fully trusts VMM
– VMM should not trust guest
– Restricts VMM from interacting with guest
• Symbiotic VMM must trust guest interfaces
• BUT it doesn’t have to use them
– Selectively enable interfaces depending on trust level
• I will examine the implications Symbiotic
virtualization has on the trust model
31
Symbiotic Complexity
• Symbiotic interfaces can increase complexity
of VMMs
– Implications for Trusted Computing platforms
• Complexity is already there
– See examples of bridging the gap
• Correct functionality does not require VMM
support
32
Evaluation
• Performance impact of Symbiotic Interfaces
• Comparison against existing interfaces
– Lines of code
– Complexity of other approaches
– Explanation of how the symbiotic functionality is not
otherwise possible
• Evaluate functionality with several example cases
• Examine how issues are addressed by design
• Also evaluating virtualization and HPC at scale
33
Implementation
• Implementation of formalized design
• Environment
– VMM: Palacios
– Host OS: Kitten
– Guest OS: Kitten and Linux
• Reasoning:
– Relatively small code size
– Familiarity with both
34
Code size
35
Symbiotic Examples
• Demonstrating symbiotic virtualization
– Symbiotic Swap
– Symbiotic Device Drivers
– Symbiotic Assists
• Made possible by a symbiotic design
36
Virtualized Memory
• VM memory model same as physical memory
– Shadow/Nested paging designed to mimic
• OS has memory set at boot time
– Exceptions
• Rare support for hot pluggable memory
• Paravirtualized memory
– Usually a large change to Guest OS
• Swap storage allows over allocation
– Can be exhausted
– Can lead to thrashing
37
Current Swap Architectures
•
In Linux, swap storage is an array of pages
–
•
Easily accessible
When a page is swapped its given an index
value
–
–
–
Points to array location
Page faults occur on page access
OS retrieves page and moves it to physical
memory
38
Symbiotic Swap
• Purpose: prevent thrashing situations
– Temporarily expand memory
• A symbiotic OS would expose swapped page map
– VMM could find swapped page with minimal effort
• Guest OS begins to thrash
– Detected by VMM
– Guest page is swapped out, but VMM copies it to free page
– Shadow memory is altered to point to swapped page
• Accesses no longer cause faults
• Thrashing ends
– VMM synchronizes swapped out page
– Next access will fault the page back in to guest memory
39
Symbiotic Swap Architecture
40
Device Drivers
• Guests often need direct device access
– High performance networks
– Driver included inside guest OS
• Self-virtualization
– Devices still require their own drivers
– Not all devices are capable
• Does not map well to virtual environments
– Migration changes underlying hardware
– Difficult to share between multiple VMs
– VMM must fully trust guest driver
41
VPIO: Virtual Passthrough IO
• Modeling-based approach to high performance I/O
virtualization for commodity devices
– Devices with no virtualization support
• VMM runs Device Model Monitor (DMM)
– Intercepts a subset of IO commands
– Maintains model of internal device state
– Transitions model state based on IO operations
• Prevents security violations
• Determines when device can be context switched
L. Xia, J. Lange, and P. Dinda, Towards Virtual Passthrough I/O on Commodity
Devices, Proceedings of the First Workshop on I/O Virtualization at OSDI
42
NE2k Device Model
43
Symbiotic Device Drivers
• VMM provides passthrough driver to guest
– Passthrough driver can include VPIO model
• Design OS to allow driver injection
• Guest OS no longer needs to include full set of drivers for all
possible hardware
• VMM can optimize driver behavior to the environment
• Drivers can be dynamically swapped as conditions change
– Passthrough network driver
– Overlay network driver
– Paravirtual driver
44
VMM Extended Services
• VMMs perform operations on OS
– Migration, suspend/resume, checkpointing
• One sided approaches are often overly complex
– VMM must account for OS behavior
– Many have not been successfully implemented inside an OS
• Ideally services are supported by both VMM and Guest OS
– VMM and guest OS share responsibility
– Each one does what is suitable to their environment
45
Symbiotic Assists
• Possible Uses
– Notifications
• Guest OS is aware of VMM events
– Optimizations
• VMM can request guest optimize itself for an operation
• Example: Migration
– Allow guest OS to optimize itself for migration
• Flush memory, freeze processes, disable devices
– Pre/Post migration notifications
• Possibly interrupt based
– A non-symbiotic OS will still work
• But won’t be optimized
46
Schedule
47
Contributions
• Bridging the Semantic Gap
– Automatic network reservations (VRESERVE)
– Virtual network services (VTL)
• Palacios
– A new VMM architecture for HPC
• Kitten
– Lightweight HPC OS
• Evaluation of virtualization in HPC at scale
48
Expected Contributions
• Formal definition of Symbiotic Virtualization
– Design of a set of symbiotic interfaces.
• Implementation of Symbiotic Virtualization
– Based on formal design
– Implemented in Palacios
– Linux and Kitten guests
• Evaluation of the Symbiotic Virtualization
– Raw performance
– Complexity comparison
49
Expected Contributions
• Example extensions
– Symbiotic Swap
• Guest OS thrashing detection
– Symbiotic Device Drivers
• Dynamic insertion of device drivers
– Symbiotic Assists
• Optimize VM operations inside guest OS
50
Related Work
• Pre-Virtualization
– Dynamically transform an OS to implement Paravirtualization
• FoxyTechnique
– Modify virtual hardware to modify guest behavior
– Does nothing to bridge the semantic gap
• Bridging the semantic gap
– Lycosid
• Security introspection
– Antfarm
• Process behavior inference
– Geiger
• Buffer cache inference
– IBMon
• Infiniband communication monitoring
51
Thank you
52