Transcript Document

API Testing:
Category Partition Method
February 21-22, 2005
© 2005 SIPTECH
www.siptech.com
The Context

SIPTECH specializes in testing and optimization of
technology platforms and software products
based on Java.

API conformance testing for several components
of a mobile handset platform.
© 2005 SIPTECH
www.siptech.com
2
API Testing

Application Programming Interface (API) is a
collection of software procedures and functions

Two main requirements in testing an API are:

To test all functionalities

To obtain near 100% code coverage
© 2005 SIPTECH
www.siptech.com
3
Test Statistics
Total tests developed / modified
across Iterative cycles
Number of Iterative cycles
Defects unearthed so far
© 2005 SIPTECH
www.siptech.com
4888
18
557
4
Problem Statement

70% Code Coverage due to Heuristic approach

Time constraint due to Iterative cycle

Multiple specifications to comply with

Inconsistencies in the work products due to
subjectivity in test creation
© 2005 SIPTECH
www.siptech.com
5
Category Partition Method

Category Partition Method (CPM) is a systematic,
specification based method that uses partitioning to
generate functional tests for complex software systems

Assumption that defects are related to value
combination of various parameters and instance
variables
© 2005 SIPTECH
www.siptech.com
6
CPM Implementation

Method Interface arguments and abstract class
state that determine function’s response are
identified

Identify categories (equivalence classes) for each
of the parameter identified
© 2005 SIPTECH
www.siptech.com
7
CPM Implementation

Identify a specific test value (choice) for each
category – min boundary conditions

Identify constraints between choices

Enumerate all choice combinations

Fill out expected results for all combinations
© 2005 SIPTECH
www.siptech.com
8
Illustration

Class
:
FileFolder

Method
:
getEntries

Return
:
Entry [ ]

Parameters:
int startPosition must be >= 1
int length must be >= 0

Global

Exception :
© 2005 SIPTECH
:
Size of Folder must be >= 0
FolderException
www.siptech.com
9
Illustration
Parameter
Category
Choice
startPosition
n<1
0
1<=n<Size
1, n<Size, n=Size-1
n>=Size
n=Size, n>Size
n<=0
n<0, 0
0<n<=Size
1, n<Size, n=Size
n>Size
n=Size+1
n<0
-1
n=0
0
n>0
n
Length
Size
© 2005 SIPTECH
www.siptech.com
10
Empirical Analysis

TCDT = Effort for TCD / No of TCD
TCDT Vs Iteration
CPM Pilot
TCDT
0.6
0.4
0.2
0
1
2
3
4
5
Component 1
0.49
0.49
0.46
0.55
0.33
Component 2
0.32
0.38
0.41
0.32
0.23
Iteration
TCD – Test Case Definition; TCDT – Test Case Definition Time
© 2005 SIPTECH
www.siptech.com
11
Empirical Analysis

TCDRT = Effort for TCD Review / No of TCD
TCDRT Vs Iteration
TCDRT
0.3
CPM Pilot
0.2
0.1
0
1
2
3
4
5
Com ponent 1
0.22
0.2
0.21
0.21
0.096
Com ponent 2
0.14
0.15
0.18
0.13
0.08
Iteration
TCD – Test Case Definition; TCDRT – Test Case Definition Review Time
© 2005 SIPTECH
www.siptech.com
12
Empirical Analysis

Code Coverage
Code Coverage Vs Iteration
Code Coverage
CPM Pilot
100
80
60
40
20
0
1
2
3
4
5
Component 1
72
74
69
72
75
Component 2
81
83
80
82
84
Iteration
© 2005 SIPTECH
www.siptech.com
13
Empirical Analysis

Cost of Testing = Efforts / No of Defects
Testing Cost
Testing Cost Vs Iteration
80
60
40
20
0
CPM Pilot
1
2
3
4
5
Component 1
24
36.8
62.3
23.4
19.5
Component 2
31.8
27
73.5
33.5
21
Iteration
© 2005 SIPTECH
www.siptech.com
14
Limitations

Does not find performance, stress related defects

Size of the test suite

Subjectivity is reduced but still persists in
identifying categories or choices
© 2005 SIPTECH
www.siptech.com
15
Future Steps

Complementary technique to limit size

Create a repository of parameters mapped to
categories and choices

Provide user interface for repository

Automate choice combination for test suite
generation
© 2005 SIPTECH
www.siptech.com
16
THANK YOU…
M. Chandramouleswaran ([email protected])
R. Dhanasekaran ([email protected])
SIP Technologies and Exports Limited
G4, Elnet Software City, Taramani, Chennai 600 113, India.
Ph : 91-44-22541473/74; Fax: 91-44-22541475
Visit us at : www.siptech.com
© 2005 SIPTECH
www.siptech.com
17