Lightweight Modeling of Java Virtual Machine Security Constraints using Alloy

Mark Reynolds BU CS511 Midterm Report March 26, 2008

• Motivation • The Model • Results • Future Work

Outline

Motivation

• Java enforces its security restrictions in three ways: – Compile time – Load time (the Bytecode Verifier) – Runtime • Runtime type checks • Array bounds checks • SecurityManager checks • How good are these checks?

A Long History of Java Exploits

• Edward Felten’s applet attacks (1997) – “Securing Java” McGraw & Felten • “Last Stage of Delirium” (Polish hacker group) (2002) – Extensive collection of exploit techniques • Defects continue to be found – Search for “jre” or “jdk” on http://cve.mitre.org

• Almost all defects are in the Bytecode Verifier

The Bytecode Verifier

• Operates on binary class files – “JVM assembly language” • Performs a superset of the set of checks performed by the Java compiler • Uses a constraint based approach – Local variable constraint – Stack depth invariance constraint – Opcode constraint – Method argument constraint – Many others

Local Variable Constraint

• Every JVM local variable is written before it is read • In the JVM implementation local variables are completely separate from the stack, unlike most other architectures (e.g. x86) • Goal: write an Alloy model that can operate on a representation of method bytecode and check the local variable constraint

An Example in Java

public int fred(int input) { int tmp; if ( input < 10 ) tmp = input*5; else tmp = input; return tmp; }

An Incorrect Variant

public int fred(int input) { int tmp; if ( input < 10 ) /* tmp = input*5; */ {} else tmp = input; return tmp; }

Bytecodes for Correct Variant

// 0 0:iload_1 // 1 1:bipush 10 // 2 3:icmpge 13 // 3 6:iload_1 // 4 7:iconst_5 // 5 8:imul // 6 9:istore_2 // 7 10:goto 15 // 8 13:iload_1 // 9 14:istore_2 // 10 15:iload_2 // 11 16:ireturn

Bytecodes for Incorrect Variant

// 0 0:iload_1 // 1 1:bipush 10 // 2 3:icmpge 13 // 3 6:iload_1 // 4 7:iconst_5 // 5 8:imul // 6 9:nop // removed the “unnecessary” write to LV2 // 7 10:goto 15 // 8 13:iload_1 // 9 14:istore_2 // 10 15:iload_2 // 11 16:ireturn

The Model

• Define the relations on a sig Instruction: – map: Int – len: Int – r: set Int – w: set Int – ubt: lone Int – cbt: lone Int – term: lone Int

Relations in the Model

map – byte offset of this instruction from start of method len – byte length of instruction r, w – sets of local variables read/written by the instruction ubt – unconditional branch targets for this instruction (e.g. goto) cbt – conditional branch targets for this instruction (e.g. icmpge 13) term – does this instruction terminate the method (e.g. ireturn)

Class File

Model Construction

Alloy Java Declarations Class2Alloy Translator Initialization Method name Body Facts Predicates Assertions

Example Initialization

one sig startup, iload_1a, bipush, icmpge, iload_1b, iconst, imul, istore_2a, goto, iload_1c, istore_2b, iload_2, ireturn extends Instruction {} } fact maps { map = startup->(-1) + iload_1a->0 + bipush->1 + icmpge->3 + iload_1b->6 + iconst->7 + imul->8 + istore_2a->9 + goto->10 + iload_1c->13 + istore_2b->14 + iload_2->15 + ireturn->16 fact terms { term = ireturn->1 } } fact lens { len = startup->1 + iload_1a->1 + bipush->2 + icmpge->3 + iload_1b->1 + iconst->1 + imul->1 + istore_2a->1 + goto->3 + iload_1c->1 + istore_2b->1 + iload_2->1 + ireturn->1 fact rs { r = iload_1a->1 + iload_1b->1 + iload_1c->1 + iload_2->2 } fact ws { w = startup->0 + startup->1 + istore_2a->2 + istore_2b->2 } fact ubts { ubt = goto->15 } fact cbts { cbt = icmpge->13 }

Relations as a table

startup iload_1 bipush 10 icmpge 13 iload_1 iconst_5 imul istore_2 goto 15 iload_1 istore_2 iload_2 ireturn map -1 0 1 3 6 7 8 9 10 13 14 15 16 len 1 1 2 3 1 1 1 1 3 1 1 1 1 1 2 1 r 1 w 0,1 term 2 2 1 ubt 15 cbt 13

Model Execution

• Must simulation operation of the JVM • “startup” instruction simulates method entry – Stores “this” and method arguments into local variables 0, 1, … • Use ordering module on State sig to keep track of readers and writers • “nextInstruction” predicate uses the following algorithm: – If there is an unconditional branch take it – Otherwise, form the set {conditional branches} U next-instruction and choose a member of that set • “solve” predicate checks that execution terminated (reached an instruction with non null value of “term”) • “checkit” assertion checks that the set of readers is a subset of the set of writers for all states

} sig State { prog: Instruction, readers: set Int, writers: set Int

Key Components

} pred nextInstruction[from, to: Instruction] { some from.ubt => ( to.map = from.ubt ) else ( ( to.map = add[from.map, from.len] ) || some bt: from.cbt { to.map = bt } ) } pred solve { some finalState : State | finalState.prog.term = 1 } assert checkit { all s: State | s.readers in s.writers

Results

• Local variable constraint satisfied => no counterexample to “checkit”, instance of “solve” found (jvm.als) • Local variable constraint violated => counterexample of “checkit” found (badjvm.als) • Website: http://cs-people.bu.edu/markreyn

Future Work

• Use cbt relation for try .. catch blocks • Add stack depth invariance constraint • Add opcode constraint