Transcript Using Binary Decision Diagrams for Combinatorial Test Design

Common Patterns in
Combinatorial Models
Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick
IBM Haifa Research Labs
1
Agenda
• Introduction & Motivation
• Pitfalls in Combinatorial Models
• Modeling Patterns
2
Introduction
• Combinatorial model = attributes, values,
restrictions
• Correctly identifying them is of the main
obstacles in practicing combinatorial
testing
• There are common pitfalls, and common
patterns that address them
3
Motivation
• Construct a “checklist” for practitioners
– Common pitfalls and common solutions
• Raise the topic for community discussion
– This is not a complete list
– More patterns are welcome
4
Common Pitfalls in
Combinatorial Modeling
Correctness
Failing to capture the intention correctly
Completeness
Omitting an important part of the test space from the model
Redundancy
Explicitly enumerating different cases that are actually equivalent
5
Modeling Patterns
• Recurring patterns we encountered in
combinatorial models
– i.e., recurring properties of the modeled test
spaces
• Address the pitfalls
• Encountered in many different models,
regardless of the domain, current level of
testing, etc.
6
Overview
7
Optional and Conditionally-Excluded
Values (completeness, correctness)
• Example I: online form with fields email
(mandatory) and home address (optional)
• Naïve model:
– Email – valid / invalid
– HomeAddr – valid / invalid
• Much better:
– Email – valid / invalid
– HomeAddr – valid / empty / invalid
8
Optional and Conditionally-Excluded
Values – cont.
• Example II: online form with email
address, home address, and “is billing by
email”
• Home address used only if not billing by
email
9
Optional and Conditionally-Excluded
Values – cont.
• Naïve model:
– Email – valid / invalid
– HomeAddress – valid / invalid
– BillByEmail – true / false
– Not allowed:
• HomeAddress == “valid” && BillByEmail
• HomeAddress == “invalid” && BillByEmail
10
Optional and Conditionally-Excluded
Values – cont.
• Much Better:
– Email – valid / invalid
– HomeAddr – valid / invalid / empty / NA
– BillByEmail – true / false
– Not allowed:
• HomeAddr != “NA” && BillByEmail
• HomeAddr == “NA” && ! BillByEmail
11
Multi-Selection
(correctness, completeness)
• Example: shopping cart system. The
shopping cart may contain meat, dairy, fish
and drinks
• Naïve model:
– Cart Item – meat / dairy / fish / drinks
12
Multi-Selection – cont.
• A better solution:
– Item 1 – meat / dairy / fish / drinks
…
– Item t – meat / dairy / fish / drinks / none
• An even better solution:
– Meat – true / false
– Dairy – true / false
– Fish – true / false
– Drinks – true / false
13
Ranges and Boundaries
(redundancy, completeness)
• Applied when values of a parameter can
be divided into ranges, such that values in
a range are equivalent with respect to the
testing scenario
14
Ranges and Boundaries – cont.
• Example I: file processing. File of up to 1
MB is handled by a first approach. File of
1-256 MB by a second, and over 256 by a
third
• Model:
– FileSize: 0 / moreThan0LessThan1 /
1 / moreThan1LessThan256 / 256 /
moreThan256
15
Ranges and Boundaries – cont.
• Example II: user updates insurance plan.
User chooses benefits for the new plan,
some may already exist.
• Model:
– Benefits – allBenefitsExisting /
someExistingSomeNew / allBenefitsNew
16
Order and Padding (completeness)
• When? When a requirement defines an
output that depends on two (or more)
inputs.
• Modeler Q: Can bugs surface if inputs are
presented in different orders ?
17
Order and Padding – cont.
• Example: expense reimbursement.
Expenses covered only if both food and
lodging expenses are submitted
• Naïve model:
– Lodging – true / false
– Food – true / false
18
Order and Padding – cont.
if (record.type == “Food” && pLodgingWasSubmitted)
{ process(record); }
if (record.type == “Lodgin” && pFoodWasSubmitted)
{ process(record); }
19
Order and Padding – cont.
• Better model:
– Lodging – true / false
– Food – true / false
– FoodBeforeLodging – true / false /
NotApplicable
20
Order and Padding – cont.
• Modeler Q II: may the implementation
depend on the records being consecutive?
• Modeler Q III: may the implementation
depend on either records being first / last?
21
Order and Padding – cont.
• Updated model:
– Lodging – true / false
– Food – true / false
– FoodBeforeLodging – true / false /
NotApplicable
– PaddingBefore – true / false / NotApplicable
– PaddingBetween – true / false / NotApplicable
– PaddingAfter – true / false / NotApplicable
22
Multiplicity and Symmetry
(redundancy)
• Applicable when:
– Multiple elements of the same type
– Their interactions with other elements are
equivalent
• Example: two hosts (x86 / PowerPC), two
storage devices (Enterprise / Mid-Range /
Low-End)
23
Multiplicity and Symmetry – cont.
• Naïve Model:
– Host1 – x86 / PowerPC
– Host2 – x86 / PowerPC
– Storage1 – Enterprise / MidRange / LowEnd
– Storage2 – Enterprise / MidRange / LowEnd
24
Multiplicity and Symmetry – cont.
• Possible Solution:
– numX86Running – 0 / 1 / 2
– numPowerPCRunning – 0 / 1 / 2
– numEnterpriseRunning – 0 / 1 / 2
– numMidRangeRunning – 0 / 1 / 2
– numLowEndRunning – 0 / 1 / 2
– Not allowed:
• numX86Running + numPowerPCRunning != 2
• numEnterpriseRunning + numMidRangeRunning +
numLowEndRunning != 2
25
Multiplicity and Symmetry – cont.
• An alternative approach:
• A tool and algorithm that supports
designating equivalent parameters
– And whether internal interactions are
equivalent
• Requires a tool that supports dynamically
changing coverage requirements.
26
Summary
• Modeling is typically the most challenging
part in applying combinatorial testing
• We listed some pitfalls, and common
patterns that address them
• There are probably many more
27
Thank You For Listening
28