Transcript Checking Inside the Black Box: Regression Fault Exposure

Checking Inside the Black Box: Regression Fault Exposure
and Localization Based on Value Spectra Differences
Tao Xie
Advisor: David Notkin Dept. of Computer Science & Engineering, University of Washington
Overview
Program State @exit point
Return
Values
Arguments
Global variables
Program State @entry point
In order to expose a fault,
•Execution: The faulty line(s) executed
•Infection: The data flow infected
•Propagation: The infection propagated
Output correct = Behavior correct?
Check inside the black box in addition
to the output in regression testing. What
to check inside?
•Assertion/Invariant: too loose
•Structural spectrum: too restrictive
Value Spectra
•Program output value spectrum (POV)
•Function value count spectrum (FVC)
•Execution-trace value spectrum (ETV)
•Function value hit spectrum (FVH) 
FVH captures stable information across versions
•Value spectra 
Program Execution =
Internal program states + transitions
Version 2
int main(int argc, char *argv[])
{
int i, j;
if (argc != 3) {
printf(“Wrong arguments!”);
return 1;
}
i = atoi(argv[1]);
j = atoi(argv[2]);
if (max(i,j) > 0){
if (max(i, j) < 10){
printf(“2”);
} else {
printf(“1”);
}
} else {
printf(“0”);
}
return 0;
}
Spectra
argv[1]
“1”
argv[2]
“7”
Profiled Entities
Output 2
3
a
b
1
a
7
b
return
FVC
Smain3_“1”_“7”|3_“1”_“7”_0 * 1, Smax1_7|1_7_7 * 2
ETV
Smain3_“1”_“7”|3_“1”_“7”_0, Smax1_7|1_7_7, , Smax1_7|1_7_7, , 
FVH
Smain3_“1”_“7”|3_“1”_“7”_0, Smax1_7|1_7_7
1
7
a
b
1
a
7
b
7
return
main
entry
state
max
entry
state
max
exit
state
max
entry
state
argc
3
argv[1]
“1”
argv[2]
“7”
return
0
1
7
7
output
2
Smain3_“1”_“7”| 3_“1”_“7”_0:
Smax1_7|1_7_7:
main:entry(argc=3,argv[1]=“1”,argv[2]=“7”),
max:entry(a=1,b=7),
exit(argc=3, argv[1]=“1”,argv[2]=“7”,return=0)
exit(a=1,b=7,return=7)
max
exit
state
main
exit
state
Program execution with input “1 7”
Regression Fault Localization based on
Value Spectra Differences
Types of value spectra differences:
•Entry-Same-eXit-Diff (ESXD)
•Entry-Diff (ED)
•Heuristic 1. If an ED’s caller execution is not ED and neither ED
nor ESXD is present between ED’s caller and ED, fault locations
are likely to be those statements executed between ED’s caller and
that ED’s call site
•Heuristic 2. If an ESXD encloses neither ED nor another ESXD,
fault locations are likely to be those statements executed within
ESXD’s function body
Differences
Version 1
int max(int a, int b)
{ if (a >= b) return a else return b;}
argc
POV
•When there is an FVH entry-exit variable
value pair in the new version but not in the
old version and the function for that value
pair exists in the old version, regression
faults are reported
FVH
#include <stdio.h>
Value spectra for the sample program with input “1 7”
Regression Fault Exposure based on
Value Spectra Differences
FVH
Function
Execution
Arguments
Values
Global variables
Test inputs
Experiments (on 7 Siemens C programs)
main (execution of test t58 on faulty version 9 of tcas program)
initialize
alt_sep_test
Non_Crossing_Biased_Climb
Inhibit_Biased_Climb
Own_Above_Threat
Non_Crossing_Biased_Descend
Inhibit_Biased_Climb
(ED)Own_Below_Threat
(ED)ALIM
Own_Above_Threat
main (execution of test t91 on faulty version 9 of tcas program)
initialize
(ESXD) alt_sep_test
Non_Crossing_Biased_Climb
Inhibit_Biased_Climb
Own_Above_Threat
ALIM
Own_Below_Threat
(ESXD)Non_Crossing_Biased_Descend
Inhibit_Biased_Climb
Own_Below_Threat
Initial
Findings:
Statistical Summary of Experiment Results
Items
Data
Items
Data
Mean(H1_ok/H1_all)
100%
Stdev (H1_ok/H1_all)
25.7%
Mean(H2_ok/H2_all)
100%
Stdev (H2_ok/H2_all)
37.6%
Mean(POV Diff/Cov)
3.5%
Stdev (POV Diff/Cov)
26.9%
Mean(FVH Diff/Cov)
10.3%
Stdev (FVH Diff/Cov)
39.1%
•POV Diff: # test executions that exhibit POV differences
•FVH Diff: # test executions that exhibit FVH differences
•Cov: # test executions that cover the faulty statements
•Hi_all: # test executions eligible to use localization heuristic i
•Hi_ok: # test executions being applied localization heuristic i
successfully
Increases the regression fault exposure probability by two times and
locates the faulty functions accurately
Issues
•Spectra evolution accommodations:
filtering and mapping in spectra
comparison
•Scalability: instrumenting modified
functions and their direct/indirect
callees
Ongoing and Future Work
•Value-spectra-based dynamic change
impact analysis for unit testing
•Modular component regression testing
•Value-spectra-based threading fault
exposure and localization
•More experimental subjects
This work was supported in part by the National Science Foundation under grant ITR 0086003.