Transcript Folie 1

BAiT:
Adaptive Test Case Execution
in Practice
Jens R. Calamé
Centrum voor Wiskunde en Informatica,
Amsterdam
Agenda
1.
2.
3.
4.
5.
Introduction
The Toolset for BAiT
An Exemplary Case: The ATM
Structure of BAiT
Conclusion
13-09-2007
DWFTT 2007, Enschede
2
Introduction
• BAiT: Behavior Adaptation in Testing
• Toolset to generate test cases and test oracles as well as
execute tests on nondeterministic systems with data
13-09-2007
DWFTT 2007, Enschede
3
Agenda
1.
2.
3.
4.
5.
Introduction
The Toolset for BAiT
An Exemplary Case: The ATM
Structure of BAiT
Conclusion
13-09-2007
DWFTT 2007, Enschede
4
BAiT: Test Case & Oracle Generation
• Test generation with TGV
(enumerative approach)
• Chaotic data abstraction
for input and output
parameters of events
• Test oracle generation for
Eclipse Prolog constraint
solver in order to select
test data
• Automatic generation of
variable names for the test
case
13-09-2007
DWFTT 2007, Enschede
5
BAiT: Test Proxy Generation
• Test Proxy for Java-based
SUTs
• Instantiation of test object
within test proxy
• Procedure-based
bidirectional
communication (publishersubscriber pattern)
• Mapping of datatypes from
specification language to
target platform
13-09-2007
DWFTT 2007, Enschede
6
BAiT: Test Execution
• Test execution of precalculated test trace
(behavior and data)
• Adaptation to the IUT's
reactions, rather than
immediate assignment of
verdict
• BAiT framework provides
adapters for user-defined
– trace selection and
– test data selection
algorithms
13-09-2007
DWFTT 2007, Enschede
7
Agenda
1.
2.
3.
4.
5.
Introduction
The Toolset for BAiT
An Exemplary Case: The ATM
Structure of BAiT
Conclusion
13-09-2007
DWFTT 2007, Enschede
8
An Exemplary Case: The ATM
ATM specification (partially)
13-09-2007
ATM test purpose (completely)
DWFTT 2007, Enschede
9
An ATM Test Case
!initPin(1234)
!getPin(1234)
?pinCorrect
!initBalance(1000)
!getAmount(100)
?retFifty(1)
?retFifty(2)
?retTwenty(2)
?retTen(1)
Photos: Wikipedia and ECB
13-09-2007
?
DWFTT 2007, Enschede

10
An ATM Test Case (2)
1. Precalculation of
possible test trace
May 25, 2007 3:06:19 PM nl.cwi.sen2.bait.steps.TraceImpl merge
FINE: Pruning planned test trace.
May 25, 2007 3:06:19 PM nl.cwi.sen2.bait.RunTest findTrace
FINER: Examining: [init],initPin(Pin),getPin(PinUser),pinCorrect,
2. Iterative data
initBalance(Bal),getAmount(Amt),[tau],retFifty(Fif),emitBankNotes
selection
May 25, 2007 3:06:19 PM nl.cwi.sen2.bait.steps.TraceImpl solve
FINEST: init(G1),initPin(G1,G2,lparam(nat(Pin))),[...]
[Pin in {[-Infinity .. Infinity]} => 1234]
3. Stepwise execution of
the trace…
FINEST: init(G1),initPin(G1,G2,lparam(nat(1234))),[...]
May 25, 2007 3:06:21 PM nl.cwi.sen2.bait.steps.TraceImpl solve
[...]
May 25, 2007 3:06:26 PM nl.cwi.sen2.bait.RunTest executeTest
INFO: Trying: [init],initPin(1234),getPin(1234),pinCorrect,
initBalance(1000),getAmount(100),[tau],retFifty(2),emitBankNotes
…until the
IUT deviates
May 25, 2007 3:06:26 PM nl.cwi.sen2.bait.RunTest
executeTest
from the
precalculated trace.
FINE: initPin(1234) -> OK
[...]
May 25, 2007 3:06:26 PM nl.cwi.sen2.bait.RunTest executeTest
FINE: retFifty(2) -> NOT OK
May 25, 2007 3:06:26 PM nl.cwi.sen2.bait.RunTest executeTest
INFO: Expected: retFifty(2), received: retFifty(1) -> Trace failed,
trying alternative.
13-09-2007
DWFTT 2007, Enschede
11
An ATM Test Case (3)
May 25, 2007 3:06:26 PM nl.cwi.sen2.bait.RunTest executeTest
INFO: Trace deviated; executed so far:
4. Pruning not yet executed part
[init],initPin(1234),getPin(1234),pinCorrect,initBalance(1000),
of test trace.
getAmount(100),[tau],retFifty(1),retTwenty(2),retTen(1),
emitBankNotes
5. Appending received
May 25, 2007 3:06:26 PM nl.cwi.sen2.bait.steps.TraceImpl merge
system output to trace.
FINE: Pruning planned test trace.
May 25, 2007 3:06:26 PM nl.cwi.sen2.bait.steps.TraceImpl merge
FINE: Adding retFifty(1) to trace stub.
[...]
May 25, 2007 3:06:26 PM nl.cwi.sen2.bait.steps.TraceImpl merge
FINE: Adding emitBankNotes to trace stub.
May 25, 2007 3:06:27 PM nl.cwi.sen2.bait.RunTest executeTest
INFO: Test finished; executed: [init],initPin(5),getPin(5),
pinCorrect,initBalance(1000),getAmount(100),[tau],retFifty(1),
retTwenty(2),retTen(1),emitBankNotes
May 25, 2007 3:06:27 PM nl.cwi.sen2.bait.RunTest executeTest
INFO: The test case ended with verdict PASS. 6. Solve test trace with
pending system output;
assign verdict or calculate
new trace.
13-09-2007
DWFTT 2007, Enschede
12
Agenda
1.
2.
3.
4.
5.
Introduction
The Toolset for BAiT
An Exemplary Case: The ATM
Structure of BAiT
Conclusion
13-09-2007
DWFTT 2007, Enschede
13
Structure of BAiT: The Tools
• BAiT: Behavior Adaptation in Testing
• Toolset to generate test cases and test oracles as well as
execute tests on nondeterministic systems with data
13-09-2007
DWFTT 2007, Enschede
14
Structure of BAiT: Execution Part
Test Oracle
•Variable class: Codec of data to constraint solver, data
instantiator and IUT
•Data instantiation algorithm(s)
Data Mapping
Constraint Solver
Data Mapping
Impl. Under Test
Data Selection
Constraint
Solver
BAiT
Execution Core
Impl.
Under
Test
Trace Selection
Test
Case(s)
13-09-2007
Behavior Mapping
Constraint Solver
Behavior Mapping
Impl. Under Test
•Trace class: trace execution management
•Step class: communication with IUT
•Trace selection from complete test graph
(IOLTS with all possible test cases for a given test purpose)
DWFTT 2007, Enschede
15
Agenda
1.
2.
3.
4.
5.
Introduction
The Toolset for BAiT
An Exemplary Case: The ATM
Structure of BAiT
Conclusion
13-09-2007
DWFTT 2007, Enschede
16
Conclusion
• Behavior Adaptation in Testing helps finding more reliable
verdicts in testing non-deterministic systems
• A prototypical framework for BAiT has been developed
procedure-based setting with atomic data
• Further work:
– Extension of framework
for other settings
– Extension of framework
to structured datatypes
– Case Study:
Mozilla Firefox
CSS subsystem
13-09-2007
DWFTT 2007, Enschede
17
Further Information
Background information
• Calamé, Jens R.; Ioustinova, Natalia; Pol, Jaco van de; Sidorova, Natalia; Data
Abstraction and Constraint Solving for Conformance Testing (APSEC 2005).
• Calamé, Jens R.; Ioustinova, Natalia; Pol, Jaco van de; Towards Automatic
Generation of Parameterized Test Cases from Abstractions. Technical Report, 2006.
The toolset/framework itself
• Calamé, Jens R.; Adaptive Test Case Execution in Practice. Technical Report, 2007.
• Calamé, Jens R.; The Internals of Adaptive Test Case Execution. Technical Report,
to appear.
All downloadable from www.calame.de/works,published.bib,date,.html
Contact:
Jens R. Calamé
CWI, Amsterdam
[email protected]
13-09-2007
DWFTT 2007, Enschede
18