Transcript ICSE 2007 Presentation

Automated Generation of Context-Aware Tests
by
Zhimin Wang (University of Nebraska–Lincoln)
Sebastian Elbaum (University of Nebraska–Lincoln)
David S. Rosenblum (University College London)
Research funded in part by the EPSRC and the Royal Society
Background
Ubiquitous Computing Systems
An important emerging class of software systems
• Context-Aware
– Execution driven by changes
in execution environment
Radio signal
strength
Location
Nearby
device IDs
• Sensed through middleware
invocation of context handlers
 Context is an additional input space that
must be explored adequately during testing
• Adaptive
– Execution must adapt to context changes
 Changes to multiple context variables
may occur simultaneously
Available
memory
Battery
level
Problem Statement
Testing Ubiquitous Systems
Discovering concurrency
faults in context-aware
ubiquitous systems
New SMS
• Failures occur frequently
during attempts to handle
multiple context changes
• Existing testing techniques have
limited effectiveness in discovering
the underlying faults
Found Wi-Fi
Additional Challenges during Testing
• Hard to control when and how to input contexts
– Middleware can introduce noise
– Interference can occur between context handlers
• Hard to define a precise oracle
– Execution differs under various vectors of context inputs
• Real environment is not available
– Too many sensors are required
• Sensed contexts can be inconsistent
– Example: The border between rooms at an exhibition
Contributions
1. CAPPs: Context-Aware Program Points
•
A model of how context affects program execution
2. CAPPs-Based Test Adequacy Criteria
•
•
Criteria for evaluating test suite effectiveness
Defined in terms of sets of test drivers
•
Sequence of CAPPs to cover
3. CAPPs-Driven Test Suite Enhancement
•
Automated exploration of variant interleavings of
invocations of context handlers
•
Schedules interleavings via special instrumentation
Application for Case Study
TourApp Released with the Context Toolkit
Visitor
Tag
TourApp
Application
Communication
Middleware
Interpreter
Widget
PDA
Registration
Widget
Demo
Widget 1
Service
Remote data
connection
Demo
Widget 2
Service
…
End
Widget
Sensor
Sensor
Sensor
Sensor
Registration
Booth
Room 1
Room 2
Exit Room
Application for Case Study
TourApp Released with the Context Toolkit
Visitor
Tag
TourApp
Application
Communication
Middleware
Interpreter
Widget
PDA
Registration
Widget
Demo
Widget 1
Service
Remote data
connection
Demo
Widget 2
Service
…
End
Widget
Sensor
Sensor
Sensor
Sensor
Registration
Booth
Room 1
Room 2
Exit Room
• Location:
Registration
Room
• Application
Response:
Pop-Up
Registration
Form
Application for Case Study
TourApp Released with the Context Toolkit
Visitor
Tag
TourApp
Application
Communication
Middleware
Interpreter
Widget
PDA
Registration
Widget
Demo
Widget 1
Service
Remote data
connection
Demo
Widget 2
Service
…
End
Widget
Sensor
Sensor
Sensor
Sensor
Registration
Booth
Room 1
Room 2
Exit Room
• Location:
Demo
Room 1
• Application
Response:
Display
Lecture
Information
Application for Case Study
TourApp Released with the Context Toolkit
Visitor
Tag
TourApp
Application
Communication
Middleware
Interpreter
Widget
PDA
Registration
Widget
Demo
Widget 1
Service
Remote data
connection
Demo
Widget 2
Service
…
End
Widget
Sensor
Sensor
Sensor
Sensor
Registration
Booth
Room 1
Room 2
Exit Room
• Power:
Low
Battery!
• Application
Response:
Confine
Display
Updates
Overview of Testing Infrastructure
Context-Aware Program (P)
Test Suite (T)
CAPPs
Identifier
Annotated
Flow Graph
Context Driver
Generator
Program
Instrumentor
P
Context
Manipulator
Test Drivers (D)
Selected Context
Adequacy Criteria
Feedback on Coverage
Achieved Coverage
And
Test Case Extension
Test Adequacy Criteria
Context Adequacy (CA)
• Test driver covering at
least one CAPP in each
type of context handler
• Examples
– { capp1, capp2 }
11
11
capp1 12
14
13
capp2
19
16
– or { capp3, capp2 }
Handler CFG
type="demo"
Handler CFG
type="power"
15
16
20
Exit
27
capp5
28
Exit
capp3 21
– or { capp2, capp1 }
26
capp4 22
23
Exit
29 capp6
30
31
32
Exit
Test Adequacy Criteria
Switch-to-Context Adequacy (StoC-k)
• Set of test drivers covering
all possible combinations
of k switches between
context handlers
• StoC-1 Example:
– { capp1, capp2 },
{ capp5, capp3 },
{ capp3, capp3 },
{ capp5, capp5 }
Handler CFG
type="demo"
Handler CFG
type="power"
11
11
capp1 12
14
13
capp2
19
16
15
26
16
20
Exit
27
capp5
28
Exit
capp3 21
capp4 22
23
Exit
29 capp6
30
31
32
Exit
Test Adequacy Criteria
Switch-with-Preempted-Capp Adequacy
(StoC-k-FromCapp)
• Set of test drivers covering
all possible combinations
of k switches between
context handlers, with each
switch exercised at every
capp
CAPP
• StoC-1-FromCapp Example:
1
– { capp1, capp2 },
{ capp3, capp2 },
{ capp4, capp5 },
{ capp2, capp3 },
{ capp5, capp1 },
{ capp6, capp3 },
{ capp3, capp3 },
{ capp5, capp5 }
Handler CFG
type="demo"
Handler CFG
type="power"
11
11
12
14
13
capp2
19
16
15
26
16
20
Exit
27
capp5
28
Exit
capp3 21
capp4 22
23
Exit
29 capp6
30
31
32
Exit
Case Study Design and Settings
TourApp
•
•
•
•
11 KLOC of Java, 4 seeded faults
Test suite of 36 end-to-end test cases
Executing test suite takes 10 minutes
Studied 4 versions:
– originalTourApp: unmodified original
 manipulatedTourApp: instrumented with calls to our
scheduler methods
– delayShortTourApp: instrumented with 1–3 seconds
random delays (sleep())
– delayLongTourApp: instrumented with 1–10 seconds
random delay (sleep())
Results: Cost
Timings with manipulatedTourApp
• Summary of study
– Execution time increases with more demanding context
coverage criteria as more context scenarios are
required
Results: Feasibility
Drivers in manipulatedTourApp
• Summary of study
– Some test drivers were not realised in the application
within the set timeouts
– Improvements in generation of D may be needed via
better flow-sensitive analysis
Results: Effectiveness
Percentage of Contextual Coverage
and Fault Detection
• Summary of study
– Coverage decreases with more powerful criteria
• But only slightly for manipulatedTourApp
– manipulatedTourApp performs the best, especially with
more powerful criteria
Related Work
• Deterministic testing of concurrent programs
(Carver & Tai, Taylor et al.)
– Concurrency intrinsic to program, not execution environment
• Metamorphic testing for context-aware applications
(Tse et al.)
– Oracles embodying metamorphic properties must be defined by
tester
• Data flow coverage criteria for context-aware applications
(Lu et al.)
– Does not support notion of CAPPs or manipulation of test
executions
• Random sleeps for test perturbation (Edelstein et al.)
– Inferior to controlled scheduling of context switches
Conclusion
• Defined the CAPPs model of how context
changes affect context-aware applications
• Defined test adequacy criteria in terms of this
model
• Created an automated technique to guide test
executions in a way that systematically explores
many interesting context change scenarios
• Demonstrated the superiority of this technique in
discovering concurrency faults
Future Work
• Investigate Additional Classes of Faults
– User interface ‘faults’
– Adaptation priority faults
– Memory leaks
• ContextNotifier and TestingEmulator
– Emulation infrastructure for testing
– (Tools and libraries from vendors are pathetic!)
• Simulation-Driven Testing
– Test case execution driven by mobility traces from
simulation runs
Questions?
http://www.ubival.org/