Transcript HyperSentry: Enabling Stealthy In

Ahmed M. Azab, Peng Ning, Zhi Wang, Xuxian Jiang
Department of Computer Science, North Carolina State University
Xiaolan Zhang
IBM T.J. Watson Research Center
Nathan C. Skalsky
IBM Systems & Technology Group
2011/3/8
ADL Meeting
1
Outline
 About SMM
 Introduction and Background
 Assumptions, Threat Model, and Security




Requirements
The HyperSentry Framework
Verifying the Integrity of the Xen Hypervisor – a Case
Study
Implementation and Experimental Evaluation
Conclusion
2011/3/8
ADL Meeting
2
About SMM - Reference
 Phrack Magazine:
 Issue #65: System Management Mode Hack: Using SMM
for “Other Purposes”
 Issue #66: A Real SMM Rootkit: Reversing and Hooking
BIOS SMI Handlers
 Duflot, Using CPU System Management Mode to
Circumvent Operating System Security Functions
 Intel Architecture Software Developers Manuals,
Volume 3: System Programming
2011/3/8
ADL Meeting
3
About SMM
 SMM: System Management Mode [wiki]
 Intel manuals:
 “The Intel System Management Mode (SMM) is
typically used to execute specific routines for power
management. … SMM operates independently of other
system software, and can be used for other purposes
too.”
2011/3/8
ADL Meeting
4
About SMM
<- rsm or reset
Real Address Mode
-> SMI (interrupt)
PE=0 or reset
PE = 1
<- rsm instruction
reset
Protected Mode
SMM Mode
-> SMI (interrupt)
VM = 0
VM = 1
<- rsm instruction
Virtual 8086 Mode
-> SMI (interrupt)
2011/3/8
ADL Meeting
5
About SMM
 Instruction RSM can be used just inside the SMM
 Paging is disabled
 16-bit mode of operation
 But all physical memory can be addressed (4GB)
 The same privileges as in Ring 0
 Interrupts are blocked in SMM (So are NMI
Interrupts)
 SMI: System Management Interrupt
 SMRAM: System Management RAM
2011/3/8
ADL Meeting
6
About SMM
 SMRAM
 SMRAM region is at 0xA0000-0xBFFFF

Video card memory-mapped base address
 SMRAM Control Register
 Bit 6 – D_OPEN
 SMBASE is redirected to SMRAM if D_OPEN is set
 Bit 4 – D_LCK
 To protect SMRAM
2011/3/8
ADL Meeting
7
About SMM
 SMI_EN register:
 Control which devices can generate an SMI
 The least significant bit specifies whether SMIs are
enable or not
 SMI_STS register:
 Keep track of which device last caused an SMI
2011/3/8
ADL Meeting
8
Introduction
 Hypervisors did not turn out to be completely secure
 Xen is used in Amazon’s EC2
 2008 Black Hat: Xen 0wning trilogy (Blue Pill)
 [part1] [part2] [part3] [code_and_demo]
 Attack Xen
 There are at least 17 vulnerabilities reported for Xen 3.x
 There are at least 165 vulnerabilities reported in
Vmware ESX 3.x
 The true challenge lies in the measurement of
hypervisor integrity at runtime
2011/3/8
ADL Meeting
9
Introducion
 Copilot uses PCI device to provide integrity
measurement
 Cannot access CPU state (e.g., CR3 register)
 HyperGuard and HyperCheck
 Rely on SMM
 None of them provide a way to trigger the integrity
measurement without alerting hypervisor
 Scrubbing Attacks
2011/3/8
ADL Meeting
10
Introduction
 Challenges
 Stealthy integrity measurement
 Verifiable measurement agent
 Deterministic and non-interruptible execution
 In-context measurement
 Attestation to the authenticity of the measurement
output
2011/3/8
ADL Meeting
11
Introduction
 HyperSentry relies on a Trust Computing Base (TCB)
composed of hardware, firmware and software
 HyperSentry is triggered by an out-of-band
communication channel
 SMI: System Management Interrupt
 IPMI: Intelligent Platform Management Interface [wiki]

May use AMT (Intel Active Management Technology) to
trigger SMI
 HyperSentry resides in the SMM
2011/3/8
ADL Meeting
12
Introduction
 SMM does not offer all the necessary contextual info
 Set the CPU to the required context
 Provide a verifiable and protected environment to run a
measurement agent in the hypervisor context
 IBM BladeCenter H chassis with HS21 XM blade
servers
 Xen
2011/3/8
ADL Meeting
13
Background
 IPMI:
 Server-oriented platform management interface
 Hardware + Firmware
 Baseboard Management Controller (BMC) on
motherboard
 SMM
2011/3/8
ADL Meeting
14
Assumptions, Threat Model, and
Security Requirements
 Assumptions:
 Equipped with an out-of-band channel
 Physically secured
 TCG’s trusted boot hardware and Trusted Platform
Module
2011/3/8
ADL Meeting
15
Assumptions, Threat Model, and
Security Requirements
 Threat Model:
 The adversary, once compromising the hypervisor, will
attempt to attack the measurement software and/or
forge measurement output
 Periodic integrity measurement

2011/3/8
Do not handle attacks that do not cause a persistent change
ADL Meeting
16
Assumptions, Threat Model, and
Security Requirements
 Security Requirements:
 (SR1) Stealthy Invocation
 (SR2) Verifiable Behave
 (SR3) Deterministic Execution
 (SR4) In-context Privileged Measurement
 (SR5) Attestable Output
2011/3/8
ADL Meeting
17
The HyperSentry Framework
VM
VM
VM
Guest (non-root) Mode
Host (root) Mode
Hypervisor
Measurement
Agent
Hardware
Virtualized Platform
2011/3/8
ADL Meeting
SMI
Handler
Trusted
Components
are Shaded
in Green
System
Management
Mode
IPMI/BMC
Remote
Verifier
18
The HyperSentry Framework
 When interrupted by the SMI…
 CPU may run in either the hypervisor (VMX root
operation) or one of the guest VM (VMX non-root
operation)
 In order to measure the integrity of hypervisor, the
measurement agent needs to access the hypervisor’s
code, data and CPU state
 When CPU runs in VMX non-root operation at SMI, all
pointers to VMX data structures are saved internally to
the CPU and cannot be retrieved via software
2011/3/8
ADL Meeting
19
The HyperSentry Framework
 Challenges
 How to detect the intercepted CPU operation mode?
 Hypervisor or guest VM?
 How to retrieve the context needed for measurement?
 E.g., CR3 and page table
 Solution
 Inject a privileged instruction to force the CPU to fall
back to the hypervisor mode
 Run the measurement agent in the same context as the
hypervisor

2011/3/8
Agent runs in a protected execution environment
ADL Meeting
20
The HyperSentry Framework
Guest VM
Execution Path
SMI
RSM
Prepare SMM fallback
Inject privileged
instruction and flush cache
Privileged instruction
Guest (non-root) Mode
Hypervisor
Host (root) Mode
VM exit
SMI
RSM
SMI
The measurement
agent
Hardware
2011/3/8
Verify the measurement
agent
Store measurement output
System Management Mode
PC (cache misses = 0)
1)
ADL Meeting
APIC (SMI on PC overflow)
21
The HyperSentry Framework
 If an interrupt is received during handling ths SMI…
 Inject another copy of the instruction at each interrupt
handler
 Using LIDT instruction to locate interrupt handler
2011/3/8
ADL Meeting
22
The HyperSentry Framework
 In-context Integrity Measurement
 Some Intel Trusted eXecution Technology late launch
registers are hidden from the SMM
 SMM is relatively slow

2 orders of magnitude slower than protected mode
 So, HyperSentry’s measurement agent runs in protected
mode
2011/3/8
ADL Meeting
23
The HyperSentry Framework
 Measurement Agent Verification
 SMI handler calculates the hash of the measurement
agent’s code
 SMI handler disables all maskable interrupts by clearing
the corresponding bit in EFLAGS register
 SMI handler modifies the IDT (physical memory)
 Malicious DMA write operation threat is handled by
verifying that the agent is included in the DMA
protected ranges provided by Intel VT-d
2011/3/8
ADL Meeting
24
The HyperSentry Framework
 Handling Multi-core Platforms
 Freeze all cores
 Only Boot Strap Processor (BSP, core 0) will execute
HyperSentry
2011/3/8
ADL Meeting
25
The HyperSentry Framework
 Is out-of-band invocation sufficient to achieve
stealthy invocation?
 Unfortunately …
2011/3/8
ADL Meeting
26
The HyperSentry Framework
VM
VM
VM
Guest (non-root) Mode
Host (root) Mode
SMI
Handler
Typical
Scenario
System
Management
Mode
Hypervisor
Hardware
BMC/IMM
Remote
Verifier
2011/3/8
ADL Meeting
27
The HyperSentry Framework
VM
VM
VM
Guest (non-root) Mode
Host (root) Mode
SMI
Handler
Attack
Scenario
System
Management
Mode
Hypervisor
Hardware
BMC/IMM
Compromised hypervisor cannot intercept SMIs. But what
if it tries to block real
2011/3/8
ADL SMIs
Meeting and generate fake ones?
Remote
Verifier
28
The HyperSentry Framework
 Can we prevent the hypervisor from blocking
SMIs?
 Not possible with existing hardware
 Solution
 Detecting fake SMIs generated by the (compromised)
hypervisor

Verifying status registers to ensure that the measurement is
invoked by the out-of-band channel
 Key reason: HW SMI and SW SMI are distinguishable
2011/3/8
ADL Meeting
29
The HyperSentry Framework
CPU
Core 0
CPU
Core n
CPU
Core 1
SMI
- All status register are non writable
- Measurement is invoked only if all
other bits are 0
- A fake SMI is easily detectable
Memory Control Hub (North Bridge)
SMI_EN
1
GPI_ROUT
0 1
ALT_GPI_SMI_EN
1
10 9
0
SMI_STS
0 …..0 1 0…….0
ALT_GPI_SMI_STS
0 ……………….0 1
GPI 0 BMC
IO Control Hub (South Bridge)
2011/3/8
Target PlatformADL
(IBM
HS21XM Blade Server)
Meeting
IPMI AMM
SSH
Remote
Verifier
30
The HyperSentry Framework
 HyperSentry requires that GPI_ROUT is configured so
that only GPI 0 can generate SMIs
 If a compromised hypervisor disable SMI by
overwriting GPI_ROUT
 Easily detected due to lack of response
2011/3/8
ADL Meeting
31
The HyperSentry Framework
 Attesting to the Measurement Output
 Challenge

Absence of a dedicated hardware for attestation
 The hypervisor controls the hardware most of time
 Solution


2011/3/8
Providing the SMRAM with a private key
Using this key to attest to the measurement results
ADL Meeting
32
The HyperSentry Framework
Ksmm-1{ Output|Nonce}
Remote
Verifier
Guest
Bootstrapping
VM
SMI handler
Attestation
request
SMM private key
Guest Mode
Hypervisor
Host Mode
Initialization
code
output
Ksmm
KAIK-1{Ksmm |Handler|Nonce}
System Management
Mode
Hardware
2011/3/8
ADL Meeting
Integrity
-1
Ksmmmeasurement
TPM
SMM
public key
33
The HyperSentry Framework
 Stealthy Invocation
 If configurations are not changed  guaranteed by hardware
 If configurations change  fake SMIs are detectable
 Verifiable Behavior
 The measurement agent is measured every time before it executes
 Deterministic Execution
 The measurement agent possesses full control over the system
 In-context privileged measurement
 Guarantee falling back to the hypervisor mode
 The measurement agent runs in the same context as the hypervisor
 Attestable output
 The measurement output is signed by a verifiable and protected key
2011/3/8
ADL Meeting
34
Implementation and Experimental
Evaluation
 IBM HS21XM blade server
 Measuring the Xen hypervisor
 End-to-end execution time: 35 ms
2011/3/8
ADL Meeting
35
Implementation and Experimental
Evaluation
 Periodical measurement:

2011/3/8
Every 8 seconds: 2.4% overhead; every 16 seconds: 1.3% overhead
ADL Meeting
36
Conclusion
 HyperSentry
 A novel framework for measuring the integrity of the
most privileged system software
 A measurement agent for the Xen hypervisor
 Low overhead
 Next step
 Measurement agent for Linux/KVM
 Verifying the hypervisor’s dynamic integrity
2011/3/8
ADL Meeting
37
Reference
 https://researcher.ibm.com/researcher/files/us-
xmeng/Azab.pptx
2011/3/8
ADL Meeting
38