Transcript Slide 1

QE Efforts Optimization using
Orthogonal Array Testing Strategy
-A Case Study
-Monica Kothari
1
Execute Smartly?
OR
Execute Endlessly?
2
Major Challenges
Scope
3
What If...
• Less no. of tests cases but still high functional
and code coverage
• Systematic scoping of test cases across the
domain with minimal manual effort
• Find new bugs which could not be found using
conventional approach
• Find bugs early in the cycle
60% Less Effort !
4
Agenda
 Complexity of the testing problem
 Geometric View of Test Cases
 Taguchi’s OATS approach
 Example on Orthogonal Array
 Why use OATS
 Comparison of Test Planning Methods
 OATS implementation
 Applying the Technique
 ROI Analysis
 What works and What Doesn’t!
5
Complexity of a Testing Problem
Full FC 15s (16)*
r
to
Ca
16
t(
d
t
Ad Upda
)
t
eQ
R
Apply Coupon(8)
Buyer
Ap
p
(16)
Temp FC
(16)
Mini-cart
(16)
)
y(8
(8)
e Item
v
o
m
e
Appl
y
Full FC
S hi p
(8)
Regular Item
(8)
Store Tag Item
(8)
Cross-sell Item (8)
Mono-Inscn Item (8)
Volmn Item
(8)
Incr-Decr Price (8)
Softgood
(8)
GC
(8)
Verify
ly
Fre
es
Ch
hip
ec
(8)
ko
ut
(8)
Figure3 Test Case Coverage Flow Diagram
6
Verify
Total
(26)
SubTotal(13)
Price
(5)
Qty
(5)
Ship Tax (9)
Discount (5)
Key Limitations
• Generating all combinations manually is error prone:
Total Combinations =
Template(5)*FCType(4)*Item(8)*Action(8)*6(Verify)
= 7,680 combinations
• Identifying optimal combinations to test without
missing any complex combinations ensuring high
functional and code coverage ensuring the confidence in
quality of product.
• Minimum testing cycles for execution and automation
development.
7
Geometric View of Test Cases
8
Orthogonal Array Testing Strategy
Orthogonal Array Testing Strategy created by Dr. Genichi Taguchi is
a proven, systematic, statistical way of testing pair-wise interactions.
 Orthogonal Array is an array of
values in which:
 Each column is an
independent variable to be
tested for interaction
represented as – “Factor”.
 Each Factor can take a
certain set of values called
“Levels”.
 Each row represents a test
case/combination.
Taguchi OATS Approach
9
OATS - Example
A
B
C
1
2
3
4
1
1
1
2
1
2
3
1
3
2
1
2
3 Values – 1,2,3
5
6
7
2
2
3
2
3
1
1
3
1
All possible cases
involving 3
parameters:
8
9
3
3
2
3
3
2
3 Parameters – A,B,C
Applying
OATS
TABLE1 Sample Array using OA
3*3*3 = 27 cases
10
Total cases = 9 which cover all
pair-wise combinations of the 3
variables.
Why Orthogonal Array Testing Strategy?
 Identify a minimum set of tests that will find all multi-mode defects (particularly serious defects)
• All single-mode defect will be found if every option is tested at least once (unit elements)
• 2-way or “Pair-wise” is combinations of 2 items (parameters) that cause a defect
• 3-way or “Tri-wise” is combinations of 3 items (parameters) that cause a defect
 “ Pair-wise” defect detection (pair-wise testing) finds most defects, ex. testing all pairs
typically finds 75% of defects Source: Kuhn
 NASA Deep Space Mission - study showed that 88% of bugs discovered using “pair-wise”
defect detection testing
 U.S. Food and Drug Administration – study showed 98% bugs discovered using “ pair-wise
” defect detection Source: 27 th NASA/ IEEE Software Engineering Workshop, NASA
Goddard Space Flight Center, 4-6 Dec, 2002.
 “ The Combinatorial Design Approach to Automatic Test Generation” Source: IEEE, Software
1996, Vol. 13, No. 5
 Baseline application under study resulted in a reduction in test plan time from 1 month to
less than 1 week
 In several experiments, “pair-wise” demonstrated good code coverage and defect detection
ability
11
Comparison of Test Planning Methods
Testing Method
Testing Effort Testing Efforts
(No. Of Tests) (TestPlanning)
Ability to Find
Faults
Requirements
Traceability (One
Factor at a Time)
Small
Small
Limited (a)
Exhaustive Testing
Very Large
Small
All Faults Found
(100%)
Deductive/Analytical
(White-Box Testing)
Medium
Very Large
Close to 100%
Random/Intuitive
(Based on Past
History)
Medium
Medium
Unpredictable
Orthogonal Array-
Small
Small
Based Robust
Testing
12
Better than (a);
Close to 100%
when used in
conjunction with
code coverage
analysis tools.
OATS Implementation - A Case Study
Both the approaches were followed and analyzed:
1. Traditional approach to generate the test spec based on
the functional spec and domain experience.
2. Orthogonal Array Testing Strategy to generate the
optimized test set using OA Tool.
13
OATS Implementation - A Case Study contd..
TE Effort: 9 mds
TE Effort: 3 mds
14
Applying the Technique
STEP 1: Identify the Factors and Values for generating the
Test Set.
STEP 2: Identify any dependency of values between
Factors to avoid incorrect combinations ex. Floating cart
is enabled only when mini-cart is enabled.
15
Applying the Technique
STEP 3: Input the parameters as Factors and Values OA
Tool that generates Orthogonal Array Test Set.
FIGURE1 Optimized Test Table using rdExpert Tool
16
Applying the Technique
 STEP 4: Analyze the test set generated to ensure that it
covers required cases.
 If not, modify the Factors and Values to generate
another set. This may be done by splitting the
Factors and getting 2 different sets.
 Add any manual cases which are not part of the
test set.
 Incorporate any boundary values for parameters
which are required to be tested.
17
Applying the Technique
STEP 5: Generate the Test Spec in required format
using the Script functionality of the Tool.
Figure2 Script for converting matrix to Test Spec
18
ROI Analysis for Release
OATS has been used extensively in many releases and off-cycles with excellent ROI:
19
What works! and What Doesn’t!
 Applying OATS manually is not advisable.
 Focusing the testing effort on the wrong area of the
application.
 Picking the wrong parameters to combine.
 Orthogonal array only
combinations and not all.
tests
the
most
optimal
 One must use their testing skills, expertise, and experience
to improve the test cases produced by these methods.
20
Execute Endlessly or Efficiently?
Execute Efficiently?
Execute Endlessly?
Use Conventional Approach
Use OATS
 Enables to select a small test set
 High functional and code cvg
 Increased overall productivity
OR
 Systematic scoping of cases
 Easy Analysis
 Optimal implementation time
 Minimal Test Execution
 Simpler and less error prone
 Useful for integration and
regression testing
21
References
[1] Orthogonal Array Testing Strategy (OATS) Technique - Jeremy M.
Harrell
[2] Orthogonally Speaking, StickyMinds. (2001):
http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType
=MAGAZINE&ObjectId=5031&tth=DYN&tt=siteemail&iDyn=2#table3go#t
able3go
[3] http://www.pairwise.org/tools.asp
[4] http://upiia.uab.es/teach/a21291/apunts/provaOO/OATS.pdf
[5] http://www.combinatorialtesting.com/clear-introductions
22