VTL: A Transparent Network Service Framework

download report

Transcript VTL: A Transparent Network Service Framework

Symbiotic Virtualization
John R. Lange
Thesis Proposal
Department of Electrical Engineering and Computer Science
Northwestern University
June 2009
• VMs are traditionally Black boxes
– Separated from the VMM by a semantic gap
– Does provide a clean interface
• Does that make sense in today’s environment?
– Cloud computing, live migration, differing architectures
– Guests should know they are in a VM
• Many reasons to bridge the gap
– Performance, Security, Monitoring, etc…
• Existing approaches don’t allow this
• Symbiotic Virtualization is an alternative to black box design
Symbiotic Virtualization
• Novel approach to designing VMMs and operating
• OS compatible with native hardware interface
• BUT also optionally exposes a software interface that
can be used by a VMM
• Essentially, the VMM can easily inspect and modify
the guest OS
– Optional and Incremental
The Semantic Gap
Thesis Statement
Palacios and Kitten
Symbiotic Virtualization
Semantic Gap
• VMM architectures are designed as black boxes
– Explicit OS interface (hardware or paravirtual)
– Internal OS state is not exposed to the VMM
• Many uses for internal state
– Performance, security, etc...
– VMM must recreate that state
• “Bridging the Semantic Gap”
• Many examples
– Virtuoso Project
– Lycosid, Antfarm, Geiger, IBMon, many others
Virtuoso Project
Bridged the semantic gap for virtual networking
Examine physical network traffic to model application behavior
Provide virtual services to unmodified OSes and Applications
Virtuoso Project Components
Sundararaj, A., Gupta, A., and Dinda, P. Increasing application performance in virtual environments through
run-time inference and adaptation. In Proceedings of the 14th IEEE International Symposium on High
Performance Distributed Computing
Gupta, A., and Dinda, P. Inferring the topology and traffic load of parallel programs running in a virtual
machine environment. In Proceedings of the 10th Workshop on Job Scheduling Strategies for Parallel Processing
Sundararaj, A., and Dinda, P. Towards virtual networks for virtual machine grid computing. In Proceedings of
the 3rd USENIX Virtual Machine Research And Technology Symposium (VM 2004)
Lange, J., Sundararaj, A., and Dinda, P. A. Automatic dynamic run-time optical network reservations. In
Proceedings of the 14th IEEE International Symposium on High Performance Distributed Computing
Lange, J. and Dinda, P. Transparent network services via a virtual traffic layer for virtual machines. In
Proceedings of the 16th International Symposium on High Performance Distributed Computing
• Overlay network for virtual machines
– Remotely distributed VMs appear connected to a LAN
– Layer 2 overlay, operates on ethernet frames
• Supports arbitrary overlay topologies, routing, and
link types
• Provides mechanisms to maximize network
• Virtual Topology and Traffic Inference
– Infers communication topology and traffic load
matrix for a VM
– Uses information from VTTIF
– Adaptively optimizes VNET overlay topology
• Automatic and dynamic network reservations
– Allows unmodified applications to use circuit
switched optical networks
• Added optical network reservation interface to
– Automatically reserves network link when VTTIF
detects traffic between two connected hosts
VTL: Transparent Network Services
• Manipulate data and signaling of connections to add
services to existing unmodified applications and OSes
– High Level transformations of Low Level traffic
– Transparency: Manipulations invisible to guest
environment (Black Box approach)
• VTL (Virtual Traffic Layer)
– A framework for creating Transparent Network Services
• Can transform TCP connections into different
Bridging the Semantic Gap
• Enables many useful features and
• However…
– Current approaches are labor intensive
• Reverse engineering an OS
– Highly specific to OS implementation
– Collected information not always accurate
Symbiotic Virtualization
• Bridging the semantic gap is hard
– Can we design a virtual environment with no gap?
• Symbiotic Virtualization
– Design both guest OS and VMM to minimize semantic gap
– 2 components
• Guest OS provides internal state to VMM
• Guest OS services requests from VMM
– Interfaces are optional
• Not required for correct operation
Thesis Statement
• I propose symbiotic virtualization, an approach to OS design that preserves the
benefits of full system virtualization, while enabling performance and
functionality benefits.
• In symbiotic virtualization, an OS targets the native hardware interface and can
run unmodified on raw hardware. However, it also exposes a software interface
that can be leveraged by a symbiotic virtualization-aware VMM.
• Both the interface and its use by the VMM are optional, but if it exists, and the
VMM uses it, the VMM and the OS can mutually benefit.
• Symbiotic virtualization is markedly different from the current virtualization
approaches, and is best considered as being on a continuum between full system
virtualization and paravirtualization.
Thesis Goals
Define and formalize Symbiotic Virtualization
Develop formal symbiotic interfaces
Implement symbiotic interfaces inside an OS
Implement set of symbiotic extensions
Use examples to evaluate the symbiotic
• OS independent embeddable VMM
– Written from scratch at NU and UNM
• Designed to be modularly linked into existing kernels
– Minimal host OS interface
– Compiles into static library
– Currently embedded: Kitten and GeekOS
• Open Source (BSD License)
– Downloaded ~1000 times
• Lead developer
Palacios Details
• Supports 32 and 64 bit environments
– Host and Guest
• Full hardware virtualization
– Currently only supports AMD extensions
– Intel VMX in process
• Supports Linux and HPC guest OSes
• Relatively small: ~28K lines
• Lightweight HPC OS from Sandia National Labs
– Designed for large scale HPC systems (Cray XT)
– Successor to Catamount and earlier lightweight kernels
• Based on Linux
– Only the necessary components
– Limited Linux ABI compatibility
• Uses Palacios for virtualization
– Embedded as a library
– VMs launch as part of job submission
• Contributing developer
Palacios as an HPC VMM
Minimalist interface:
– Does not require extensive host OS features
– Easily embedded into even small kernels
Full system virtualization:
– Does not require guest OS changes
– Runs existing kernels without any porting
• Kitten, Catamount, Cray CNL, and IBM’s CNK
Contiguous memory preallocation:
– Preallocates guest memory as a physically contiguous region
– Vastly simplifies the virtualized memory implementation
– Deterministic performance for most memory operations
Passthrough resources and resource partitioning:
– Host resources are easily mapped directly into a guest environment
– Provides access to high performance devices, with existing device drivers, with no
virtualization overhead.
Low noise:
– Minimizes the amount of OS noise injected by the VMM layer.
– No internal timers and no accumulated deferred work.
Symbiotic Virtualization in HPC
• HPC environments are well suited to symbiotic techniques
• Full trust of the software stack
– Fewer security concerns
• Specific hardware configurations
– Limited number of devices
• Constrained problem space
– Small number of applications
• Implementations can be very specific
• Environments are much smaller
– Internal OS state is simpler than a general purpose OS
• At large scale performance impact is dramatic
– Large impetus to optimize VMM and OS
HPC Performance Example
• Guest OS behavior can differ widely
– Must optimize for specific OSes and applications
• Example:
– Catamount and Compute Node Linux
• 2 HPC OSes
– Process switching implementation
• CNL swaps page tables
• Catamount does not
– Nested and shadow page tables have very different performance
– Evaluated with 2 HPC benchmarks
• 3 configurations (Native, Shadow Paging, Nested Paging)
• Running on RedStorm Development Cages (Cray XT)
HPCCG Benchmark
Compute Node Linux
CTH Benchmark
Compute Node Linux
• At large scale minor performance problems become large
– Very important to minimize any performance overhead introduced
• VMM needs to know about guest internals
– Should modify behavior for each guest environment
– Which paging method to use depends on guest
• Inference is not desirable in HPC environment
– Unacceptable performance overhead
– Convergence time
– Mistakes have large consequences
• Symbiotic approach is very appealing
Symbiotic Virtualization
• Definition based on formalization
– Formalized interfaces
• Two types of interfaces
– Passive information interface
• VMM can read guest OS state
– Functional interface
• VMM can send requests to guest OS
• Neither required for OS to function correctly
– Symbiotic OS can run on hardware
– Non-symbiotic OS can run on symbiotic VMM
– Can be implemented incrementally
Passive Interface
• Formalize the interface for bridging the semantic gap
– Ideally removes the gap
• Internal state already exists but it is hidden
– Existing tools try to recreate this data in the VMM
• Symbiotic Interface:
– Structure internal OS state in a way that is easily parsed
• Semantically rich
– Expose OS state to the VMM
• Easily accessible
Example interface
• Linux process list
– Organized in a series of lists
– Scattered throughout kernel address space
– Lots of information included inside
• Priority, memory map, open file descriptors, etc
• Symbiotic Interface
– Collect task information in standard location
– Organize information to be easily parsable
• Reserved memory page that holds pointers to high priority
• List of CR3 values that should be cached
Functional Interface
• Mechanism for OS to expose functionality to
– Guest OS services VMM requests
• Possible interfaces
– Guest OS notifications
– VMM can force explicit upcalls
– Iterator based system
Initial Functional Interface
• Partial initial test implementation
– Prosnitz and Xia
• Implemented inside GeekOS and Palacios
• Iterator based
– Modelled on RPCs
• New VMM/OS interaction model
• Traditional virtualization assumptions no
longer true
– No longer a black box
• Some new issues to be addressed
– Trust
– Design Complexity
Symbiotic Trust Model
• Current Architectures: unidirectional trust
– Guest OS fully trusts VMM
– VMM should not trust guest
– Restricts VMM from interacting with guest
• Symbiotic VMM must trust guest interfaces
• BUT it doesn’t have to use them
– Selectively enable interfaces depending on trust level
• I will examine the implications Symbiotic
virtualization has on the trust model
Symbiotic Complexity
• Symbiotic interfaces can increase complexity
of VMMs
– Implications for Trusted Computing platforms
• Complexity is already there
– See examples of bridging the gap
• Correct functionality does not require VMM
• Performance impact of Symbiotic Interfaces
• Comparison against existing interfaces
– Lines of code
– Complexity of other approaches
– Explanation of how the symbiotic functionality is not
otherwise possible
• Evaluate functionality with several example cases
• Examine how issues are addressed by design
• Also evaluating virtualization and HPC at scale
• Implementation of formalized design
• Environment
– VMM: Palacios
– Host OS: Kitten
– Guest OS: Kitten and Linux
• Reasoning:
– Relatively small code size
– Familiarity with both
Code size
Symbiotic Examples
• Demonstrating symbiotic virtualization
– Symbiotic Swap
– Symbiotic Device Drivers
– Symbiotic Assists
• Made possible by a symbiotic design
Virtualized Memory
• VM memory model same as physical memory
– Shadow/Nested paging designed to mimic
• OS has memory set at boot time
– Exceptions
• Rare support for hot pluggable memory
• Paravirtualized memory
– Usually a large change to Guest OS
• Swap storage allows over allocation
– Can be exhausted
– Can lead to thrashing
Current Swap Architectures
In Linux, swap storage is an array of pages
Easily accessible
When a page is swapped its given an index
Points to array location
Page faults occur on page access
OS retrieves page and moves it to physical
Symbiotic Swap
• Purpose: prevent thrashing situations
– Temporarily expand memory
• A symbiotic OS would expose swapped page map
– VMM could find swapped page with minimal effort
• Guest OS begins to thrash
– Detected by VMM
– Guest page is swapped out, but VMM copies it to free page
– Shadow memory is altered to point to swapped page
• Accesses no longer cause faults
• Thrashing ends
– VMM synchronizes swapped out page
– Next access will fault the page back in to guest memory
Symbiotic Swap Architecture
Device Drivers
• Guests often need direct device access
– High performance networks
– Driver included inside guest OS
• Self-virtualization
– Devices still require their own drivers
– Not all devices are capable
• Does not map well to virtual environments
– Migration changes underlying hardware
– Difficult to share between multiple VMs
– VMM must fully trust guest driver
VPIO: Virtual Passthrough IO
• Modeling-based approach to high performance I/O
virtualization for commodity devices
– Devices with no virtualization support
• VMM runs Device Model Monitor (DMM)
– Intercepts a subset of IO commands
– Maintains model of internal device state
– Transitions model state based on IO operations
• Prevents security violations
• Determines when device can be context switched
L. Xia, J. Lange, and P. Dinda, Towards Virtual Passthrough I/O on Commodity
Devices, Proceedings of the First Workshop on I/O Virtualization at OSDI
NE2k Device Model
Symbiotic Device Drivers
• VMM provides passthrough driver to guest
– Passthrough driver can include VPIO model
• Design OS to allow driver injection
• Guest OS no longer needs to include full set of drivers for all
possible hardware
• VMM can optimize driver behavior to the environment
• Drivers can be dynamically swapped as conditions change
– Passthrough network driver
– Overlay network driver
– Paravirtual driver
VMM Extended Services
• VMMs perform operations on OS
– Migration, suspend/resume, checkpointing
• One sided approaches are often overly complex
– VMM must account for OS behavior
– Many have not been successfully implemented inside an OS
• Ideally services are supported by both VMM and Guest OS
– VMM and guest OS share responsibility
– Each one does what is suitable to their environment
Symbiotic Assists
• Possible Uses
– Notifications
• Guest OS is aware of VMM events
– Optimizations
• VMM can request guest optimize itself for an operation
• Example: Migration
– Allow guest OS to optimize itself for migration
• Flush memory, freeze processes, disable devices
– Pre/Post migration notifications
• Possibly interrupt based
– A non-symbiotic OS will still work
• But won’t be optimized
• Bridging the Semantic Gap
– Automatic network reservations (VRESERVE)
– Virtual network services (VTL)
• Palacios
– A new VMM architecture for HPC
• Kitten
– Lightweight HPC OS
• Evaluation of virtualization in HPC at scale
Expected Contributions
• Formal definition of Symbiotic Virtualization
– Design of a set of symbiotic interfaces.
• Implementation of Symbiotic Virtualization
– Based on formal design
– Implemented in Palacios
– Linux and Kitten guests
• Evaluation of the Symbiotic Virtualization
– Raw performance
– Complexity comparison
Expected Contributions
• Example extensions
– Symbiotic Swap
• Guest OS thrashing detection
– Symbiotic Device Drivers
• Dynamic insertion of device drivers
– Symbiotic Assists
• Optimize VM operations inside guest OS
Related Work
• Pre-Virtualization
– Dynamically transform an OS to implement Paravirtualization
• FoxyTechnique
– Modify virtual hardware to modify guest behavior
– Does nothing to bridge the semantic gap
• Bridging the semantic gap
– Lycosid
• Security introspection
– Antfarm
• Process behavior inference
– Geiger
• Buffer cache inference
– IBMon
• Infiniband communication monitoring
Thank you