Transcript Image processing algorithm regression testing framework

Image processing algorithm
regression testing framework
Soumik Ukil
Testing Ground Rules
 Objectives:
 Testing is the process of executing a program
with the intent of finding errors
 A good test case is one that has a high
probability of finding an error
 A successful test is one that uncovers an error
 However, testing cannot show the absence
of defects
 Unit Testing:
 Testing to determine that individual program
modules perform to specification
 Basic units of software tested in isolation
 Regression Testing:
 Selective retesting to detect faults introduced
during modification of system
 Should be performed after any changes in
software like bug fixes
Motivation

Projects like NETT and BRP




Complex algorithms which run on hundreds of
datasets
Source is being constantly modified to fix bugs etc
Manual testing infeasible
Need automated regression testing
Testing Strategies

Dynamic Analysis



Exercise the software in its execution environment
Black Box
 Treat the system as black -box, ie no
knowledge of internal structure
 Only need to know legal input and expected
output
White Box
 Depends on Internal structure of program
 Test all paths through code
Testing Strategies
 Static Analysis
 Code Reviews
 Walkthroughs
 Inspections
Tests on image data
 Gold-standard data used as basis for
correctness of output
 Black box tests:
 Already know 'correct' output
 Generate data from algorithm
 Define tests that compare the two sets of data
Tests on image data

Generic Tests:



Check output is of correct datatype
Check image dimensions
Specific tests:



Depends on application being tested
For airway tree validation we may want to compare
distance between branchpoints
For Lung Segmentation we need to compare
volumes and/or distance between contours
Objectives of test framework

Different applications:



Regression testing after any changes
Validation of data
Flexible:


Testers only plug in specific tests
Test data generation and reporting of test results
taken care by framework
Implementation

Built on top of PyUnit:

Part of standard Python library for Python 2.1 and
later

Based on JUnit, a proven testing architecture

Allows creation of user defined tests, aggregation
into suites, running tests in textual or GUI mode
Writing tests with PyUnit:

Basic building blocks called Test Cases



Fixtures



A set-up and tear down method for each test case
Many different test cases can use the same fixture
Test Suites



Created by deriving from base class unittest.TestCase
An instance of a TestCase class is an object that can completely run
a single test method
Test case instances can be grouped together according to the
features they test
All tests can be executed together as part of a suite
Test Runner


A class whose instances run tests and report results
Can be used in text or GUI mode
Example:
class LungSegTestCase(unittest.TestCase):
def initialize(self,fname1,fname2):
A = AnaFile.AnaFile()
# Anafile object for reading images
self.data,hdr=A.read(fname1)
# segmentation result
self.ref_data,ref_hdr = A.read(fname2) # reference mask
def setUp(self):
self.labels=[20,30]
# left and right lung labels
def tearDown(self):
self.data = None
self.ref_data= None
def test1(self):## test method names begin 'test*'
"""Test to see if mask image datatype is uint8."""
self.assertEquals(self.data.typecode(),'b')
def test2(self):
"""Test to check that correct labels are present in mask file."""
for n in self.labels:
errormsg="label" + str(n) + "missing"
self.assertNotEquals(sum(ravel(equal(self.data, n))),0,errormsg)
Example:
class LungSegTestCase(unittest.TestCase):
def initialize(self,fname1,fname2):
A = AnaFile.AnaFile()
# Anafile object for reading images
self.data,hdr=A.read(fname1)
# segmentation result
self.ref_data,ref_hdr = A.read(fname2) # reference mask
def setUp(self):
self.labels=[20,30]
# left and right lung labels
def tearDown(self):
self.data = None
self.ref_data= None
def test1(self):## test method names begin 'test*'
"""Test to see if mask image datatype is uint8."""
self.assertEquals(self.data.typecode(),'b')
def test2(self):
"""Test to check that correct labels are present in mask file."""
for n in self.labels:
errormsg="label" + str(n) + "missing"
self.assertNotEquals(sum(ravel(equal(self.data, n))),0,errormsg)
Example:
class LungSegTestCase(unittest.TestCase):
def initialize(self,fname1,fname2):
A = AnaFile.AnaFile()
# Anafile object for reading images
self.data,hdr=A.read(fname1)
# segmentation result
self.ref_data,ref_hdr = A.read(fname2) # reference mask
def setUp(self):
self.labels=[20,30]
# left and right lung labels
def tearDown(self):
self.data = None
self.ref_data= None
def test1(self):## test method names begin 'test*'
"""Test to see if mask image datatype is uint8."""
self.assertEquals(self.data.typecode(),'b')
def test2(self):
"""Test to check that correct labels are present in mask file."""
for n in self.labels:
errormsg="label" + str(n) + "missing"
self.assertNotEquals(sum(ravel(equal(self.data, n))),0,errormsg)
Application dependent tests:

For lung segmentation we use an area overlap
measure to test correctness:

Check that value is above a given threshold to
'pass' test
Similar tests can be defined with distance
measures between contours
User has to plug-in appropriate tests


Flow diagram:
Sample configuration file:
[DEFAULT]
# Directory of gold-standard data
Refdir = ../ValidationData
#Directory of data generated by algorithm
Maskdir = ../NewData
# Output file extension
MaskExt = mask.img
# Name of logfile directory
Logdir = ../logfiles
[ALGORITHM]
# Dir. of raw image data to run algorithm on
Inputdir = /home/xyz/lung/Data
# Path to executable for algorithm
Execpath = ../lung
# Input file extension
InputExt = img
Reporting test results:

Textual output directed to log files
Success:
--------------------------------------------------------------------Ran 6 tests in 88.337s
OK
Failure:
FAIL: Left lung pixel count test (slice by slice).
---------------------------------------------------------------------Traceback (most recent call last):
File "lungsegtests.py", line 101, in test6
self.failIf(fail==1,errormsg)
File "unittest.py", line 264, in failIf
if expr: raise self.failureException, msg
AssertionError: significant segmentation mismatch for left lung on slices:
[458, 462]
Status:

Testing for Lung segmentation/smoothing has
been implemented with the following tests:





Type checking of output masks
Checking that all expected labels are present
Slice by slice area comparison for both lungs
Can be used by segmentation algorithms on
ANALYZE images which produce labeled
masks
Framework can be extended to handle other
formats like airway tree definitions etc
References:
1.
http://pyunit.sourceforge.net
2.
http://www.xprogramming.com/testfram.htm
Documentation and source code:
CVS repository: pulmonary/TestFrameWork