KVM/ARM: The Design and Implementation of the Linux ARM
Download
Report
Transcript KVM/ARM: The Design and Implementation of the Linux ARM
KVM/ARM: The Design and
Implementation of the Linux
ARM Hypervisor
Christoffer Dall Department of Computer Science Columbia University [email protected] Jason Nieh
Department of Compouter Science Columbia University [email protected]
32091700 김해천
ARM
~1.2 billion
~300 million
ARM
ARM Server
ARM Network infrasturcture
http://www.v3.co.uk/v3-uk/news/2397366/arm-aims-at-networkinfrastructure-with-many-core-socs-and-open-software-stack
Key Challenges
No PC-standard on ARM
Hypervisor Layering in software stack
Above part
HyperOne, Xen, PikeOS, OKL4,
Hyper V, Vmware ESX
Lower part
KVM, VirtyalBox, Virtual PC,
Parallels, BlueStacks
ARM Virtualization Extensions
Provides virtualization in 4 key areas:
CPU Virtualization
Memory Virtualization
Interrupt Virtualization
Timer Virtualization
ARM Virtualization Extensions
CPU Virtualization
Hyp mode was introduced as a trap and emulate mechanism to support
virtualization
User
User
Kernel
Kernel
Hyp
H/w
System call, page fault
To reduce virtualization overhead
ARM Virtualization Extensions
memory virtualization
Hardware support to virtualize physical memory: stage 2 Page Tables
ARM Virtualization Extensions
Interrupt virtualization
One distributor in a system, but each CPU core has a cpu Interface
Distributor is used to configure the GIC
CPU interface is used to acknowledge(ACK) and to signal End-OfInterrupt(EOI)
Interrupt can be configured to trap to either Hyp or Kernel mode
Trap to kernel : avoiding the overhead of going through Hyp mode
Trap to Hypervisor : hypervisor retain control, but big cost
VGIC
GIC v2.0 include H/W virtualization (VGIC)
Virtual CPU interface, List Register
PPI: Private Peripheral Interrupts
SPI: Shared Peripheral Interrupts
SGI: Soft Generated Interrupt
GIC : Generic Interrupt Controller
ARM Virtualization Extensions
Interrupt virtualization
Generic Interrupt Controller : Trapping Interrupt in Hyp Mode
Vm
3) Emulate Virtual Interrupt By signal
Cumbersome & Expensive
Hypervisor
2) trap
1) interrupt
H/W
ARM Virtualization Extensions
Interrupt virtualization
Generic Interrupt Controller (V2.0) , Virtual GIC : Trapping Interrupt in Kernel Mode
Vm
Hypervisor
2) trap
1) interrupt
H/W
Good
ARM Virtualization Extensions
Timer virtualization
ARM define the Generic Timer Architecture
The timers used by the hypervisor cannot be directly configured and manipulated by guest OSes.
Such timer accesses from guest OS would need to trap to Hyp mode, incurring additional overhead
counter
ARM provides ☞
Timer 0
CPU 0
Timer 1
CPU 1
Timer 2
CPU 2
Timer 3
CPU 3
Virtual counter
Virtual Timer 0
Virtual CPU 0
Virtual counter
Virtual Timer 1
Virtual CPU 1
Virtual counter
Virtual Timer 2
Virtual CPU 2
Virtual counter
Virtual Timer 3
Virtual CPU 3
Accessible from Hyp mode
Accessible from VMs
Hypervisor Architecture
KVM/ARM builds on KVM and leverages existing infrastructure in the Linux kernel
Bare metal hypervisor(xen) vs KVM/ARM
ARM platform designs are non-standard ways by different manufactures
Samsung exynos, qualcomm snapdragon, Apple A series
But, Linux is supported across almost all ARM platform ☞ by integrating KVM/ARM with Linux
PL0 User
PL1 Kernel
PL2 Hyp
Linux kernel KVM
Hypervisor Architecture
Split-mode Virtualization
Running KVM/ARM in Hyp mode implies running the Linux kernel in Hyp mode
This is problematic
Low-level architecture dependent code in Linux is written to work in kernel mode
Running the entire kernel in Hyp mode would adversely affect native performance
Kernel mode
Hyp mode
Kernel
Kernel
Hypervisor Architecture
Split-mode Virtualization
KVM/ARM introduces split-mode virtualization
It runs across different privileged CPU mode to take advantage offered by each CPU mode
Two components, the lowvisor and the highvisor
Lowvisor takes advantage of the H/W virtualization support available in Hyp mode
Set up the correct execution context by configuration of the H/W
Enforce protection and isolation between different execution context
Switch from a VM execution context to the host execution, vice versa
Provide a virtualization trap handler
Highvisor can directly leverage existing Linux functionality
Scheduler, kernel data structure, locking, memory allocation functions
Kernel
OS
mode Kernel
Hyp
mode
Hypervisor
High
visor
Handles High level
Functionality
Low
visor
Handles Low level
Functionality
Hypervisor Architecture
Split-mode Virtualization
Switching between a VM and the highvisor
OS Kernel
Kernel mode
Run VM
Hypervisor
Highvisor
Trap
Hyp mode
VM
Lowvisor
Trap
Hypervisor Architecture
Split-mode Virtualization
Switching between a VM and the highvisor
OS Kernel
Kernel mode
Function call
Hypervisor
Highvisor
Trap
Hyp mode
VM
Trap
Lowvisor
As a result, split mode virtualization incurs a double trap cost in switching to and from the highvisor
Hypervisor Architecture
CPU Virtualization
Context switch register
during world-switch
physical H/W state associated
with the hypervisor and its host
kernel is persistent across
running VMs
Performs trap and
emulate on sensitive
instruction and when
accessing H/W state
Hypervisor
VM
ARM
Virtualized cpu
trap
Controlled by the Hypervisor
S/W in the VM must have persistent
access to same register state as
S/W running on the physical CPU
Hypervisor Architecture
Memory Virtualization
KVM/ARM provides memory virtualization by enabling Stage-2 translation
When running in a VM
Completely transparent to the VM
The highvisor manages the Stage-2 translation page tables to only allow access to memory
allocated for a VM
Other accesses will cause stage-2 page faults which trap to the hypervisor
Stage-2 translation is disabled when running in the highvisor and lowvisor
Hypervisor Architecture
Memory Virtualization
Configuring page tables is a high level Functionality
OS Kernel
Kernel mode
Hyp mode
Configures Stage-2
Page Tables
Hypervisor
Highvisor
Lowvisor
VM
Hypervisor Architecture
Memory Virtualization
LowVisor has hardware access as it runs in Hyp Mode
OS Kernel
Kernel mode
Hyp mode
Hypervisor
VM
Highvisor
Lowvisor
Enables Stage-2
Translation
Hypervisor Architecture
Memory Virtualization
get_user_pages()
OS Kernel
Hypervisor
VM
Highvisor
Kernel mode
Hyp mode
Lowvisor
Disables Stage-2
Translation
Page fault
Hypervisor Architecture
Interrupt Virtualization
When running in a VM or Host & highvisor
However, VM must receive notifications in the form of virtual interrupt from emulated
devices
All, H/W interrupt processing is done in the host by using Linux’s existing interrupt handling func
KVM/ARM uses the VGIC
Multicore guest Oses musts be able to send virtual IPIs
Kernel
mode
OS
Kernel
to others
VM
High
visor
Hypervisor
Hyp
Trap
mode
Low
visor
Trap
H/W
Hypervisor Architecture
Timer Virtualization
KVM/ARM leverage ARM’s H/W virtualization features of the generic timer
Unfortunately, due to architectural limitations, the virtual timers cannot directly
raise virtual interrupts, but always raise hardware interrupt, which trap to the
hypervisor
KVM/ARM detects when a Virtual timer expires Injects
a corresponding virtual interrupt to the VM
Kernel
mode
KVM/ARM performs all hardware ACK and EOI operations
OS
Kernel
VM
VM
High
visor
Prepare
Low
visor
Hyp
mode
Hypervisor
trap
timer
Enable virtual
timer
Experimental Setup
http://www.cs.columbia.edu/~cdall/pubs/KVMARM_talk.pdf
Experimental Setup
http://www.cs.columbia.edu/~cdall/pubs/KVMARM_talk.pdf
Experimental Results
Table 3 presents costs of virtualization using KVM/ARM on ARM and KVM x86 on x86
Measured in cycle units
Hypercall
- Cost of two
world switch
Trap
- Cost of switching the
h/w mode from the
into the cpu mode
bg
bg
bg
- Saving & restore VGIC
state is quite expensive on
ARM.
- x86 provides H/W support
Experimental Results
Figure 3,4 show normalized performance for running lmbench in a VM versus Host
Caused by updating the
run-queue clock
KVM/ARM has less overhead
than KVM x86 fork & exec
UP: uni-processor
SMP:symmetrical multi-processing
Repeatedly sending an IPI Cost of KVM is higher
than KVM/ARM
Because this require tapping to the hypervisor on
x86 but not on ARM
Experimental Results
Figure 5,6 show normalized performance for running application workloads
More mature KVM x86 system has significantly higher virtualization overheads,
KVM/ARM’s split-mode virtualization design allows it to leverage ARM H/W support
with comparable performance to traditional hypervisor
Experimental Results
Figure 7 shows normalized power consumption of using virtualization
As well as ARM, mac air’s i7 is one of Intel’s more power optimized processors
Both workloads are not CPU bound & the power consumption
is not significantly affected by the virtualization layer
http://www.linux-kongress.org/2010/slides/KVM-Architecture-LK2010.pdf
http://gigglehd.com/zbxe/4440629
http://www.arm.com/
http://www.cs.columbia.edu/~cdall/pubs/KVMARM_talk.pdf
http://www.cs.columbia.edu/~cdall/pubs/KVMARM_talk.pdf
esc2014chhypforarmv7-140829040738.pdf
kvm/arm experiences building the linux arm hypervisor
Virtual CPU
Interface
Virtual CPU
Interface
VIRQ
ACK