Transcript KVMonitor
Efficient VM Introspection in KVM
and
Performance Comparison with Xen
Kenichi Kourai
Kousuke Nakamura
Kyushu Institute of Technology
Intrusion Detection System (IDS)
IDSes detect attacks against servers
Monitor the systems and networks of servers
Alert to administrators
Recently, attackers attempt to disable IDSes
Before they are detected
This is easy because IDSes are running in servers
detect
IDS
server
intrude
IDS Offloading
Offloading IDSes using virtual machines (VMs)
Run a server in a VM
Execute IDSes outside the VM
Prevent IDSes from being compromised
Can be provided as a cloud service
Cloud providers can protect users' VMs
VM
monitor
IDS
In-VM monitoring
IDS
VM
IDS offloading
VM Introspection (VMI)
A technique for monitoring VMs from the outside
Memory introspection
Obtain raw memory contents and extract OS data
Disk introspection
Obtain raw disk data and interpret a filesystem
Network introspection
VM
Obtain packets only from/to VMs
IDS
packets
???
memory
???
disk
network
Performance of VMI
Performance has not been reported in detail
No performance comparison
E.g., VMwatcher
[Jiang+ CCS'07]
Implemented in Xen, QEMU, VMware, and UML
Reported only for UML
E.g., EXTERIOR
[Fu+ VEE'13]
Implemented in KVM and QEMU
No difference due to using memory dump
Performance data is important
For user's selection of virtualization software
The Purpose of This Work
Performance comparison among virtualization
software in terms of VMI
Target: Xen and KVM
Widely used open source virtualization software
System architecture is different
process
VM
VM
VM
hypervisor
OS
Xen
KVM
Implementation for KVM
No efficient implementation of VMI for KVM
Several studies have been done for KVM
The implementation details are unclear
LibVMI
KVM
[Payne+ '11]
supports VMI for both Xen and
The performance of memory introspection is too low
in KVM
Optimized for Xen
KVMonitor
We have developed an efficient VMI tool for KVM
Execute an IDS as a process of the host OS
Provide functions for introspecting memory, disks,
and NICs in QEMU
VM
offload
IDS
KVMonitor
disk
monitor
host OS
NIC
memory
KVM module
QEMU
Memory Introspection (1/2)
Difficult to efficiently introspect QEMU's memory
LibVMI obtains memory contents from QEMU
KVMonitor shares VM's physical memory with
QEMU via a memory file
Access As a memory-mapped file
Enable direct memory introspection
IDS
VM
KVMonitor
QEMU
VM's physical
memory
VM's physical
memory
memory file
Memory Introspection (2/2)
IDSes usually access OS data using virtual
addresses
KVMonitor translates virtual addresses into
physical addresses
Look up the page table for address translation
Introspect the CR3 register using QMP
IDS
page
table
CR3
VM
KVMonitor
QEMU
VM's physical
memory
VM's physical
memory
memory file
Disk/Network Introspection
KVMonitor introspects VM's disks via the network
block device (NBD)
Interpret the qcow2 format in the NBD server
Interpret the filesystem in the host OS
KVMonitor captures packets from a tap device
IDS
KVMonitor
NBD
disk
image
file
VM
NBD server
host OS
QEMU
tap
network
Transcall with KVMonitor
We have ported Transcall
KVM
[Iida+ '11]
for Xen to
Enable offloading legacy IDSes without any
modifications
Consist of a system call emulator and a shadow
filesystem
Including the proc filesystem
Analyze OS data by memory introspection
IDS
Transcall
KVMonitor
VM
analyze
QEMU
Experiments
We examined that KVMonitor achieved
Efficient memory introspection
No impact on memory performance of a VM
Effective IDS offloading
PC
VM
CPU: Intel Xeon E5630 (12 MB L3 cache)
Memory: 6 GB DDR3 PC3-8500
HDD: 250 GB SATA
NIC: gigabit Ethernet
Hypervisor: KVM 1.1.2
Host OS: Linux 3.2.0
CPU: 1
Memory: 512 MB
Disk: 20 GB (ext3)
Guest OS: Linux 2.6.27
KVMonitor vs. LibVMI
We measured the performance of memory
introspection
KVMonitor
LibVMI
KVMonitor was
32x faster than LibVMI
read (GB/s)
Copy VM's physical memory
by 4KB
12
10
9.6
fast
8
6
4
2
0
0.3
Why is LibVMI so slow?
LibVMI has to issue a QMP command for each
memory access
Memory contents are transferred from QEMU to
LibVMI
IDS
QMP
LibVMI
LibVMI
VM
IDS
QEMU
KVMonitor
VM's
memory
VM's
memory
VM
memory
file
KVMonitor
QEMU
VM's
memory
In-VM Memory Performance
Doesn't using a memory file affect memory
performance of a VM?
memory file
Using a memory file was
memory
file
VM
VM
QEMU
QEMU
VM's
memory
VM's
memory
memory file
malloc
throughput (GB/s)
as efficient as malloc
10
malloc
8.6 8.5
8
6.6 6.3
6
4
2
0
read
write
KVMonitor vs. In-VM Access
KVMonitor was faster than
in-VM memory access
KVMonitor
Due to virtualization
overhead
10
VM
IDS
KVMonitor
VM's
memory
memory
file
QEMU
VM's
memory
read (GB/s)
8
6
4
2
0
In-VM
9.6
8.6
fast
Offloading Legacy IDSes (1/3)
Tripwire
Check filesystem integrity in disks
We added, deleted, and modified files
Offloaded Tripwire detected changed files
Rule Name
... Added Removed Modified
Monitor Filesystems
1
1
1
Total Objects scanned: 67082
Total violations found: 3
VM
Tripwire
DB
disk
Offloading Legacy IDSes (2/3)
Snort
Inspect network packets
We performed portscans from another host
Offloaded Snort detected portscans
[**] [1:1421:11] SNMP AgentX/tcp request [**]
[Classification: Attempted Information Leak] ...
01/28-10:47:13.406931 192.168.0.68:47962 -> 192.168.0.81:705
Snort
rule sets
packets
VM
portscan
Offloading Legacy IDSes (3/3)
Chkrootkit
Detect rootkits using ps, netstat, and file inspection
We tampered with ps and netstat in a VM
Offloaded chkrootkit detected tampered commands
ROOTDOR is ’/’
Checking ’ps’...INFECTED
Checking ’netstat’...INFECTED
:
execute
ps
VM
disk
chkrootkit
netstat
...
ps
netstat
Cross-view Diff (1/2)
A technique for detecting hidden malware
Compare the results of VMI and in-VM monitoring
The difference means the existence of hidden
malware
C is hidden
cross-view diff
engine
A B C D ...
VM
monitor
IDS
A B D ...
IDS
Cross-view Diff (2/2)
We tampered with ps in a VM
A hidden process was detected as malicious
We tampered with netstat in a VM
A hidden port was detected as a backdoor
ps
netstat
PID TTY
TIME CMD
1 ?
00:00:00 init
2 ?
00:00:00 kthreadd
:
PID TTY
TIME CMD
2 ?
00:00:00 kthreadd
:
Proto ... Local Address ...
tcp
0.0.0.0:5900
tcp
0.0.0.0:22
:
Proto ... Local Address ...
tcp
0.0.0.0:22
:
results from offloaded commands
results from in-VM commands
KVMonitor vs. Xen
We compared the performance of VMI between
KVM and Xen
Using a VMI tool for Xen
Memory: standard library
Disk: loopback mount
Network: tap device
Dom0 (VM)
disk image
file
tap
IDS
Hypervisor: Xen 4.1.3
Dom0 OS: Linux 3.2.0
VM: fully virtualized
VM
libxenctrl
hypervisor
Memory Introspection
We measured read throughput
Copy VM's physical memory
by 4KB
12
48x faster than Xen
read (GB/s)
KVMonitor was
10
KVM
Xen
9.6
8
fast
6
4
2
0.2
0
VMI
Why is Xen so slow?
Xen has to map each memory page
It cannot map all the pages in advance
It takes time proportional to the number of pages
KVMonitor can read a pre-mapped file
VM
IDS
IDS
libxenctrl
KVMonitor
map
Xen
VM's
memory
memory
file
KVMonitor
Kernel Integrity Checking
We measured the execution time of the kernel
integrity checker
KVM
Read the code area
Translate virtual to
physical addresses
118x faster than Xen
224
200
time (ms)
KVMonitor was
250
Xen
150
100
fast
50
0
1.9
Why is the speedup so larger?
The speedup in the real IDS was much larger
48x (simple benchmark)
118x (kernel checker)
Due to address translation
In Xen, the access cost of the page table is high
Only 8 bytes are read after memory mapping
VM
IDS
libxenctrl
map &
read
simple benchmark
VM
IDS
libxenctrl
map &
read
real kernel checker
Disk Introspection
We measured the execution time of Tripwire
For two formats of disks
KVM
raw and qcow2
9.4 9.2
Comparable to Xen
The difference between
formats was larger
time (min)
10
KVMonitor was
Raw was faster than qcow2
8
Xen
7.5 7.5
6
4
fast
2
0
raw
qcow2
Network Introspection
We measured the packet loss rate in Snort
Send many packets as
fast as possible
more lightweight
than Xen
Dom0 suffered from
virtualization overhead
12
packet loss rate (%)
KVMonitor was
KVM
Xen
10.4
10
8
6
4
2
0
6.2
fast
Chkrootkit
We measured the execution time of chkrootkit
KVM
KVMonitor was
60
1.6x faster than Xen
2x slower than in-VM
Due to system call traps
55
50
time (sec)
Efficient memory
introspection
No virtualization
overhead
Xen
40
35
fast
30
18
20
21
10
0
Offloading
in-VM
Related Work
VMI tools
Livewire [Garfinkel+ NDSS'03] for VMware
XenAccess [Payne+ ACSAC'07] for Xen
Shm-snapshot for LibVMI
[Xu+ PDL'13]
Take a VM's memory snapshot in shared memory
It takes 1.4 seconds for 3 GB
Volatility
[Walters '07]
A memory forensics framework
VMI for KVM is enabled by a Python adapter,
PyVMI from LibVMI
Conclusion
KVMonitor
Achieve efficient VM introspection (VMI) in KVM
32x faster than existing LibVMI
Performance comparison with Xen
118x faster at maximum
Chkrootkit was 1.6x faster
Future work
Comparison with other virtualization software
Integration with LibVMI