Future of WebServices testing (40min)
Download
Report
Transcript Future of WebServices testing (40min)
Advanced Modeling
Model Based Test Generation
and
Abstract state machine Language
Keith Stobie
Test Architect
XML Web Services
Microsoft Corporation
What’s a Model?
A model:
Is an abstraction or simplified
representation of the system
from a particular perspective
Supports investigation, discovery,
explanation, prediction, or
construction
May be expressed as a
description, table, graphical
diagram, or quantitative
mathematical model
Is not necessarily comprehensive
From: PNSQC 2002 “Modeling: A Picture's Worth
Words” Copyright (c) 2002, Quality Tree Software, Inc.
AsmL1000
Modelling
SASQAG 16Jan03
© 2003 Microsoft Corporation, All Rights Reserved
2
Models in Everyday Life
AsmL Modelling
SASQAG 16Jan03
© 2003 Microsoft Corporation, All Rights Reserved
Inspired by :“Modeling: A Picture's
Worth 1000 Words” Copyright (c)
2002, Quality Tree Software, Inc.
3
Examples of Models
Flow Charts
Data Flow Diagrams
Entity- Relationship Diagrams
State Diagrams
Deployment Diagrams
Class Diagrams
Use Cases
Activity Diagrams
State Transition Tables
AsmL Modelling
SASQAG 16Jan03
© 2003 Microsoft Corporation, All Rights Reserved
Inspired by :“Modeling: A Picture's
Worth 1000 Words” Copyright (c)
2002, Quality Tree Software, Inc.
4
Model Based Test Generation
Formal Model
set of
Generator
Test Generation
Directives
Test Cases
• sequence of stimuli
• expected responses
NOT just model based input generators
(pair-wise, random data, grammars, etc.)
NOT modeling tools (e.g. UML)
AsmL Modelling
SASQAG 16Jan03
© 2003 Microsoft Corporation, All Rights Reserved
5
Modeling
Product Idea
/ Specification
What product
are you
building?
Modeling
Are you
building the
right product?
Validation
Specification
AsmL
Refinement
Verification
Implementation
C/C++/C#
Are you building
the product right ?
AsmL Modelling
SASQAG 16Jan03
© 2003 Microsoft Corporation, All Rights Reserved
6
Benefits (GOTCHA)
Testing pilot amounted to 10 person months,
including the GOTCHA learning curve.
This time is less than the time used to
originally test the subsystem (12 person
months).
revealed 2 documentation changes defects
15 of the 18 defects found by the original
function test effort would also have been found
by this pilot test effort.”
From: Using a model-based test generator to
test for standard conformance
AsmL Modelling
SASQAG 16Jan03
© 2003 Microsoft Corporation, All Rights Reserved
7
Benefits
(TestMaster)
“In a typical application of this approach, test
engineer productivity has increased by a
factor of five to ten over conventional manual
approaches (Figure 5)”.
From: Model Based Testing
AsmL Modelling
SASQAG 16Jan03
© 2003 Microsoft Corporation, All Rights Reserved
8
FSM: Start State & action
From: Intelligent Test Automation by Harry Robinson
Software Testing & Quality Engineering (STQE)
magazine. Sept/Oct 2000
Start state (Entry Point)
Action
Invert_File
Selection
FilesShown: 0
FileSelect: NotSelected
ExitPoint: False
AsmL Modelling
SASQAG 16Jan03
© 2003 Microsoft Corporation, All Rights Reserved
9
State
Invert_File
Selection
other
States
start
Create_File
FilesShown: 1
FileSelected: Selected;
ExitPoint: False
Delete_File
End State (Exit Point)
SASQAG 16Jan03
AsmL Modelling
© 2003 Microsoft Corporation, All Rights Reserved
FilesShown: 0
FileSelected: NotSelected
ExitPoint: True
10
Simple FSM Model
AsmL Modelling
SASQAG 16Jan03
© 2003 Microsoft Corporation, All Rights Reserved
11
File-in-a-directory Model
Start State (Entry Point)
start
State
Invert_File
Selection
FilesOpened 0
FileSelected: NotSelected
ExitPoint: False
Action
Create_File
Invert_File
Selection
Delete_
File
FilesOpened 1
FileSelected: NotSelected;
ExitPoint: False
FilesOpened: 1
FileSelected: Selected
ExitPoint: False
Invert_File
Selection
Delete_File
exit
FilesOpened 0
End State (Exit Point)
FileSelected: NotSelected
AsmL Modelling
ExitPoint: True
SASQAG 16Jan03
© 2003 Microsoft Corporation, All Rights Reserved
12
Walk Every State-Change
Create
Invert Selection
Invert Selection
Delete
AsmL Modelling
SASQAG 16Jan03
© 2003 Microsoft Corporation, All Rights Reserved
13
File-in-a-directory Model
Start State (Entry Point)
start
State
FilesOpened 0
FileSelected: NotSelected
ExitPoint: False
Invert_File
Selection
Action
Create_File
C3
Delete_
File
Invert_File
Selection
C1
FilesOpened 1
FileSelected: NotSelected;
ExitPoint: False
C2
FilesOpened: 1
FileSelected: Selected
ExitPoint: False
C4
Invert_File
Selection
Delete_File
SASQAG 16Jan03
AsmL Modelling
exit
End State (Exit Point)
© 2003 Microsoft Corporation, All Rights Reserved
FilesOpened 0
FileSelected: NotSelected
ExitPoint: True
14
Test Every Walk < 5 Steps
A: Invert Selection
B: Create; Delete
C: Create; Invert Selection; Invert Selection; Delete
AsmL Modelling
SASQAG 16Jan03
© 2003 Microsoft Corporation, All Rights Reserved
15
Test Model Toolkit
AsmL Modelling
SASQAG 16Jan03
Source: Testing Internet Explorer:
© 2003 Microsoft Corporation, All Rights Reserved
From Underdog To Best-In-Class Browser
16
Test Model Toolkit - Impact
Used by over 20 teams at MS
Time to automate fell by as much as 88%
BizTalk - Took 1 week to generate a set of test cases
that took 8 weeks by hand
Code coverage increased by as much as 50%
One subteam in Shell – In 2 weeks increased code
coverage from 20% to 75%, while the number of test
cases increased from ~75 to ~2000.
Many spec and implementation bugs are
uncovered in the course of model creation
AsmL Modelling
SASQAG 16Jan03
Source: Testing Internet Explorer:
© 2003 Microsoft Corporation, All Rights Reserved
From Underdog To Best-In-Class Browser
17
The AsmL modeling toolkit
Abstract state machine Language is a
powerful modeling language
AsmL provides
Authoring via MS Word and XML
Simulation and interoperability via COM and .NET
Automatic test oracle
Semi-automatic test case generation
Integration with Microsoft’s test platform
AsmL Modelling
SASQAG 16Jan03
© 2003 Microsoft Corporation, All Rights Reserved
18
AsmL: Getting Started, ‘sort’ example
var A as Seq of Integer = []
Non-determinism
Swap()
choose i in indices(A),
j in indices(A)
where i < j and A(i) > A(j)
A(i) := A(j)
Parallelism
A(j) := A(i)
A = [2,3,1]
A = [1,3,2]
A = [2,1,3]
A = [1,2,3]
sort()
step until fixpoint
swap()
AsmL Modelling
SASQAG 16Jan03
© 2003 Microsoft Corporation, All Rights Reserved
19
AsmL: Predefined Data Types
Sets
menu = {"ham","cheese", "blt"}
Sequences
random = [3, 6, 7, 1, 5] //as Seq of Integer
Maps
directory as Map of String to Integer =
{“police” -> 112 , “info” -> 118}
Tuples
nameAndAge = (“John”,39) //(String,Integer)
AsmL Modelling
SASQAG 16Jan03
© 2003 Microsoft Corporation, All Rights Reserved
20
The cool part: Asml’s high-level-ness
comprehension to pick 1 of the oldest filters in a
Most Recently Used (MRU) cache.
class CacheFilter
var lastAccessed as DateTime
var MruSet as Set of CacheFilter = {}
let victim = any i | i in MruSet where
(forall j in MruSet holds
// j access time newer or equal i
(Compare(j.lastAccessed,
i.lastAccessed) >= 0))
AsmL Modelling
SASQAG 16Jan03
© 2003 Microsoft Corporation, All Rights Reserved
21
XML & XPath
Attribute Axis
<DocumentRoot>
<ParentElement Attrib1=1 attrib2=2 >
<childElementA/>
Child Axis
<childElementB/>
</ParentElement>
</DocumentRoot>
XPath:
/DocumentRoot/ParentElement/childElementB
/DocumentRoot/ParentElement/@attrib2
AsmL Modelling
SASQAG 16Jan03
© 2003 Microsoft Corporation, All Rights Reserved
22
XPath Model – axis nodes
We interpret an axis as producing either
1) all of the children of a given node or
2) all of the attribute children of a given node.
EvaluateAxis (a as Axis, c as Context) as Seq of Node
let x = ContextNode(c)
match a
Child
: return NodeChildren(x)
Axis.Attribute : return NodeAttributes(x)
_
: error "EvaluateAxis: internal error"
a=Child
Context=ParentNode
Returns: childElementA childElementB
AsmL Modelling
SASQAG 16Jan03
© 2003 Microsoft Corporation, All Rights Reserved
23
XPath model – node test
An axis path (for example, "child::para") has a twostep evaluation.
The axis ("child") is used to generate a list of
matching nodes,
which is subsequently filtered to include only those
nodes that pass the given node test ("para").
ap as AxisPath:
let nodes = EvaluateAxis(ap.axis, c)
return
[nodes(i - 1) | i in [1..Length(nodes)]
where PassesNodeTest(ap.test,
Context(nodes, i))]
AsmL Modelling
SASQAG 16Jan03
© 2003 Microsoft Corporation, All Rights Reserved
24
Using models for test
Model
Provides expected
results for
Generates
User
Info
Test Cases
Test Oracle
Pass
No pass
Provides
actual results for
Are run by
Implementation
AsmL Modelling
SASQAG 16Jan03
© 2003 Microsoft Corporation, All Rights Reserved
25
Tools & Techniques
Tool&Language
Test Technique
State Walk
Test Case Gen
Simulation
Test Oracle
Model Coverage
via random Test
Case Gen
TMT
FSM
ASML
ASM
Queries
Test Properties
e.g. all Nodes => FSM
all Edges
=> state walk
Dynamic
Model Manager
Traversal
currently
manual –
ASML test case
generation with
random paths feedback
AsmL Modelling
SASQAG 16Jan03
© 2003 Microsoft Corporation, All Rights Reserved
26
AsmL-based test harness
Any
client
Side-by-side runs
Discrepancies
automatically
flagged
Test harness
AsmL
model
Implementation
under test
AsmL Modelling
SASQAG 16Jan03
© 2003 Microsoft Corporation, All Rights Reserved
27
AsmL test case generation - 1
Solves two problems
How to find “interesting” parameter
values and combinations
How to find “interesting” call sequences
Approach
Use information derived from the model
to optimize the number of test cases
AsmL Modelling
SASQAG 16Jan03
© 2003 Microsoft Corporation, All Rights Reserved
28
AsmL test case generation - 2
Divide state space according to state predicates
System “covers” the state predicates
Driven by input generator (parameter selection) or
Driven by enabledness (call sequences)
Tester provides inputs, gives interesting state
predicates, directs the search
AsmL Modelling
SASQAG 16Jan03
© 2003 Microsoft Corporation, All Rights Reserved
29
The dual role of ASM models
Testing harness
AsmL
Model M
Call next action
IUT
Get state
view
Guided
generation
Reduce
(Param
Select
with
Feedback)
FSM
S
Pass
Is S
valid according
to M?
FSM
tool
AsmL
Model M
view
Fail: witness
Test cases
view
AsmL Modelling
SASQAG 16Jan03
© 2003 Microsoft Corporation, All Rights Reserved
30
AsmL test case generation - 3
Generate only as many test cases as needed to
have full coverage of the specification
Generator
Input
Model
Feedback
Coverage criteria
Branch: Every “branch” of the spec must be
taken
Modified condition: independent atomic
conditions
Off-by-One: off-by-one errors
Hyperstate coverage: Every abstract state
must be visited AsmL Modelling
SASQAG 16Jan03
© 2003 Microsoft Corporation, All Rights Reserved
31
XPath Model coverage
via random Param Test Generation
Model Statistics:
1,003,638 tests generated
(115Mb file)
Only 37 tests are needed
to cover all 113
(reachable 104+9)
manually
branches!
undecided, 5
dead, 36
Generator
covered, 104
covered, 9
AsmL Modelling
SASQAG 16Jan03
Model Branch Coverage
© 2003 Microsoft Corporation, All Rights Reserved
32
AsmL Test (asmlt)
asmlT can be used to:
1. Find interesting sequences of method calls
finds the shortest route in a generated FSM that uses every
edge to traverse the states
2. Find interesting parameters for each method call
Access Driven Filtering (ADF)
you write a predicate (boolean function) to decide if a given
parameter is interesting
3. Perform a conformance test
Compare Model with Implementation
AsmL Modelling
SASQAG 16Jan03
© 2003 Microsoft Corporation, All Rights Reserved
33
Good reference for
existing tools
References
Model-Based Test Generation Tools, Alan Hartman,
AGEDIS CONSORTIUM
http://www.agedis.de/documents/ModelBasedTestGenerationTools_cs.pdf
Finite State Model-Based Testing on a Shoestring, Harry Robinson
http://www.model-based-testing.org/shoestring.htm
Using a model-based test generator to test for standard conformance
E. Farchi, et al IBM SYSTEMS JOURNAL, VOL 41, NO 1, 2002
http://researchweb.watson.ibm.com/journal/sj/411/farchi.pdf
Model Based Testing, L. Apfelbaum, et al, Software Quality Week
Conference in May, 1997.
http://www.model-based-testing.org/sqw97.pdf
Modeling for Testers: A Picture’s Worth 1000 Words,
Elisabeth Hendrickson, Quality Tree Software, Inc.
http://www.raddue.com/pnsqc/Pnsqc02.pdf
AsmL Modelling
SASQAG 16Jan03
© 2003 Microsoft Corporation, All Rights Reserved
34
More References
Abstract State Machine Language, Microsoft Research
http://research.microsoft.com/fse/asml/
The AsmL Test Generator tool, Microsoft Research
http://research.microsoft.com/fse/AsmL/doc/AsmLTester.html
Testing Internet Explorer:
From Underdog To Best-In-Class Browser
Jason Taylor, Microsoft Corporation at StarWest 2002
Intelligent Test Automation, Harry Robinson
Software Testing & Quality Engineering (STQE) magazine Sept/Oct 2000
http://www.model-based-testing.org/robinson.pdf
Confessions of a Modeling Bigot: Parts III-IV
Michael Corning , ASPToday Dec02, Jan03
http://www.asptoday.com/content.asp?id=2043
© 2003 Microsoft Corporation. All rights reserved.
This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
AsmL Modelling
SASQAG 16Jan03
© 2003 Microsoft Corporation, All Rights Reserved
35