Transcript Computer Vision

Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Computer Vision – Lecture 2
Binary Image Analysis
25.10.2012
Bastian Leibe
RWTH Aachen
http://www.vision.rwth-aachen.de/
[email protected]
Announcements
• Please subscribe to the lecture on the Campus system!
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer


Important to get email announcements and L2P access!
Bachelor students please also subscribe
B. Leibe
2
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Binary Images
• Just two pixel values
• Foreground and background
• Regions of interest
Slide credit: Kristen Grauman
B. Leibe
3
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Uses: Industrial Inspection
R. Nagarajan et al. “A real time marking inspection scheme
B. Leibe
for semiconductor industries“, 2006
4
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Uses: Document Analysis, Text Recognition
Handwritten digits
Natural text (after detection)
Scanned documents
B. Leibe
5
Source: Till Quack, Martin Renold
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Uses: Medical/Bio Data
Source: D. Kim et al., Cytometry 35(1), 1999
B. Leibe
6
Uses: Blob Tracking & Motion Analysis
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Frame Differencing


Source: Kristen Grauman
Background Subtraction


Source: Tobias Jäggli
B. Leibe
7
Uses: Shape Analysis, Free-Viewpoint Video
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Visual Hull Reconstruction
Silhouette
Blue-c project, ETH Zurich
Medial axis
B. Leibe
8
Uses: Intensity Based Detection
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
• Looking for dark pixels…
fg_pix = find(im < 65);
Slide Credit: Kristen Grauman
B. Leibe
9
Uses: Color Based Detection
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
• Looking for pixels within a certain color range…
fg_pix = find(hue > t1 & hue < t2);
Slide Credit: Kristen Grauman
B. Leibe
10
Issues
• How to demarcate multiple
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
regions of interest?


Count objects
Compute further features per object
• What to do with “noisy”
binary outputs?


Holes
Extra small fragments
Slide Credit: Kristen Grauman
B. Leibe
11
Outline of Today’s Lecture
• Convert the image into binary form
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer

Thresholding
• Clean up the thresholded image

Morphological operators
• Extract individual objects

Connected Components Labeling
• Describe the objects

Region properties
B. Leibe
12
Image Source: D. Kim et al., Cytometry 35(1), 1999
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Thresholding
• Grayscale image  Binary mask
• Different variants

One-sided

Two-sided

Set membership
1, if F i, j   T
FT i, j   
0, otherwise
1, if T1  F i, j   T2
FT i, j   
0, otherwise
1, if F i, j  Z
FT i, j   
0, otherwise
B. Leibe
13
Image Source: http://homepages.inf.ed.ac.uk/rbf/HIPR2/
Selecting Thresholds
• Typical scenario
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer

Separate an object from a distinct background
• Try to separate the different grayvalue distributions



Partition a bimodal histogram
Fit a parametric distribution (e.g. Mixture of Gaussians)
Dynamic or local thresholds
• In the following, I will present some simple methods.
• We will see some more general methods in Lecture 6…
B. Leibe
14
A Nice Case: Bimodal Intensity Histograms
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Ideal histogram,
light object on
dark background
Actual observed
histogram with
noise
Source: Robyn Owens
B. Leibe
15
Not so Nice Cases…
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
• How to separate those?
Two distinct modes
Overlapping modes
Multiple modes
• Threshold selection is difficult in the general case



Domain knowledge often helps
E.g. Fraction of text on a document page ( histogram quantile)
E.g. Size of objects/structure elements
Source: Shapiro & Stockman
B. Leibe
16
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Global Binarization [Otsu’79]
• Search for the threshold T that minimizes the withinclass variance within of the two classes separated by T
2
2
2
¾wi
(T)
=
n
(T)¾
+
n
(T)¾
1
2
t hi n
1
2 (T)
where
©
ª
n1 (T) = j I (x ;y) < T j;
©
ª
n2(T) = j I (x ;y) ¸ T j
• This is the same as maximizing the between-class
variance between
2
2
2
¾bet
(T)
=
¾
¡
¾
ween
wi t hi n (T)
2
= n 1 (T)n 2 (T) [¹ 1 (T) ¡ ¹ 2 (T)]
B. Leibe
17
Algorithm
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Precompute a cumulative grayvalue histogram h.
For each potential threshold T
1.) Separate the pixels into two clusters according to T
2.) Look up n1, n2 in h and compute both cluster means
2
3.) Compute ¾bet
ween
Choose
£ 2
¤
T = argmax ¾bet ween (T)
¤
T
B. Leibe
18
Local Binarization [Niblack’86]
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
• Estimate a local threshold within a small neighborhood
window W
W
TW  W  k   W
where k [-1,0] is a user-defined parameter.
TW
What is the hidden
assumption here?
Effect:
W
B. Leibe
19
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Effects
Global threshold selection
(Otsu)
Original image
Local threshold selection
(Niblack)
B. Leibe
21
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Additional Improvements
• Document images often contain a smooth gradient
Try to fit that gradient with a polynomial function
Original image
Fitted surface
Shading compensation
Source: S. Lu & C. Tan, ICDAR’07
B. Leibe
Binarized result
22
Surface Fitting
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
• Polynomial surface of degree d
f ( x, y ) 
d
b
i  j 0
i, j
i
xy
j
• Least-squares estimation, e.g. for d=3 (m=10)
1 x 0 y0 x 02 x 0 y0

2
1
x
y
x
 1 1 1 x1 y1


2
1 x n yn x n x n yn
Ab  I
Solution with
y03   b0  I 0  pseudo-inverse:
  
3 
y1   b1  I 1  b  ( AT A) 1 AT I

,
   
   Matlab (using SVD):
3 
yn  bm  I n 
bI \A
B. Leibe
23
Surface Fitting
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
• Iterative Algorithm
1.) Fit parametric surface to all points in region.
2.) Subtract estimated surface.
3.) Apply global threshold (e.g. with Otsu method)
4.) Fit surface to all background pixels in original region.
5.) Subtract estimated surface.
6.) Apply global threshold (Otsu)
Initial
guess
Refined
guess
7.) Iterate further if needed…
• The first pass also takes foreground pixels into account.


This is corrected in the following passes.
Basic assumption here: most pixels belong to the background.
B. Leibe
24
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Result Comparison
Original image
Global (Otsu)
Local (Niblack)
Polynomial
+ Global
Source: S. Lu & C. Tan, ICDAR’07
B. Leibe
26
Outline of Today’s Lecture
• Convert the image into binary form
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer

Thresholding
• Clean up the thresholded image

Morphological operators
• Extract individual objects

Connected Components Labeling
• Describe the objects

Region properties
B. Leibe
27
Image Source: D. Kim et al., Cytometry 35(1), 1999
Cleaning the Binarized Results
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
• Results of thresholding often still contain noise
• Necessary cleaning operations


Remove isolated points and small structures
Fill holes
 Morphological Operators
B. Leibe
28
Image Source: D. Kim et al., Cytometry 35(1), 1999
Morphological Operators
• Basic idea
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer


Scan the image with a structuring element
Perform set operations (intersection, union)
of image content with structuring element
Matlab:
>> help strel
• Two basic operations


Dilation (Matlab: imdilate)
Erosion (Matlab: imerode)
• Several important combinations

Opening (Matlab: imopen)
Closing (Matlab: imclose)

Boundary extraction

B. Leibe
29
Image Source: R.C. Gonzales & R.E. Woods
Dilation
• Definition
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer


“The dilation of A by B is the set
of all displacements z, such that
ˆ and A overlap by at least one
( B)
z
element”.
B1
A
A  B1
ˆ is the mirrored version of B,
(( B)
z
shifted by z)
• Effects
If current pixel z is foreground, set all
pixels under (B)z to foreground.
 Expand connected components
 Grow features
 Fill holes

B. Leibe
B2
A  B2
30
Image Source: R.C. Gonzales & R.E. Woods
Erosion
• Definition
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer

“The erosion of A by B is the set
of all displacements z, such that
( B) z is entirely contained in A”.
B1
A
1
• Effects
If not every pixel under (B)z is
foreground, set the current pixel z
to background.
 Erode connected components
 Shrink features
 Remove bridges, branches, noise

B. Leibe
B2
2
31
Image Source: R.C. Gonzales & R.E. Woods
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Effects
Dilation with circular
structuring element
Original image
Erosion with circular
structuring element
B. Leibe
32
Image Source: http://homepages.inf.ed.ac.uk/rbf/HIPR2/
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Effects
Dilation with circular
structuring element
Original image
Erosion with circular
structuring element
B. Leibe
33
Image Source: http://homepages.inf.ed.ac.uk/rbf/HIPR2/
Opening
• Definition
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer

Sequence of Erosion and Dilation
• Effect
is defined by the points that
are reached if B is rolled around
inside A.
 Remove small objects,
keep original shape.

B. Leibe
34
Image Source: R.C. Gonzales & R.E. Woods
Effect of Opening
• Feature selection through size
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
of structuring element
Opening with small
structuring element
Original image
Thresholded
Image Source: http://homepages.inf.ed.ac.uk/rbf/HIPR2/ B. Leibe
Opening with larger
structuring element
35
Effect of Opening
• Feature selection through shape
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
of structuring element
Input Image
Opening with circular
structuring element
• How could we have extracted the lines?
B. Leibe
36
Image Source: http://homepages.inf.ed.ac.uk/rbf/HIPR2/
Closing
• Definition
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer

Sequence of Dilation and Erosion
• Effect
is defined by the points that
are reached if B is rolled around
on the outside of A.
 Fill holes,
keep original shape.

B. Leibe
37
Image Source: R.C. Gonzales & R.E. Woods
Effect of Closing
• Fill holes in thresholded image
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
(e.g. due to specularities)
Original image
Thresholded
Closing with circular
structuring element
Size of structuring
element determines
which structures are
selectively filled.
B. Leibe
38
Image Source: http://homepages.inf.ed.ac.uk/rbf/HIPR2/
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Example Application: Opening + Closing
Original image
Opening
Closing
Structuring
element
Eroded image
Dilated image
B. Leibe
39
Source: R.C. Gonzales & R.E. Woods
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Application: Blob Tracking
Absolute differences from frame to frame
Slide credit: K. Grauman
B. Leibe
40
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Thresholding
Slide credit: K. Grauman
B. Leibe
41
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Eroding
Slide credit: K. Grauman
B. Leibe
42
Morphological Boundary Extraction
• Definition
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer

First erode A by B, then subtract
the result from the original A.
 ( A)  A  ( A  B)
• Effects

If a 33 structuring element is used,
this results in a boundary that is
exactly 1 pixel thick.
B. Leibe
43
Source: R.C. Gonzales & R.E. Woods
Morphology Operators on Grayscale Images
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
• Dilation and erosion are typically performed on binary
images.
• If image is grayscale: for dilation take the neighborhood
max, for erosion take the min.
Original
Slide credit: Kristen Grauman
Dilated
B. Leibe
Eroded
44
Outline of Today’s Lecture
• Convert the image into binary form
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer

Thresholding
• Clean up the thresholded image

Morphological operators
• Extract individual objects

Connected Components Labeling
• Describe the objects

Region properties
B. Leibe
45
Image Source: D. Kim et al., Cytometry 35(1), 1999
Connected Components Labeling
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
• Goal: Identify distinct regions
Binary image
Sources: Shapiro & Stockman, Chandra
Connected components
labeling
B. Leibe
46
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Connected Components Examples
Source: Pinar Duygulu
B. Leibe
47
Connectedness
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
• Which pixels are considered neighbors?
4-connected
Source: Chaitanya Chandra
8-connected
B. Leibe
48
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Sequential Connected Components
Slide credit: J. Neira
B. Leibe
49
Sequential Connected Components (2)
• Process the image from left to
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
right, top to bottom:
1.) If the next pixel to process is 1
i.) If only one of its neighbors
(top or left) is 1, copy its label.
ii.) If both are 1 and have the
same label, copy it.
iii.) If they have different labels
 Copy the label from the left.
 Update the equivalence table.
iv.) Otherwise, assign a new label.
• Re-label with the smallest of equivalent
labels
Slide credit: J. Neira
B. Leibe
50
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Application: Segmentation of a Liver
Slide credit: Li Shen
B. Leibe
51
Outline of Today’s Lecture
• Convert the image into binary form
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer

Thresholding
• Clean up the thresholded image

Morphological operators
• Extract individual objects

Connected Components Labeling
• Describe the objects

Region properties
B. Leibe
52
Image Source: D. Kim et al., Cytometry 35(1), 1999
Region Properties
• From the previous steps, we can
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
obtain separated objects.
• Some useful features can be
extracted once we have connected
components, including





Area
Centroid
Extremal points, bounding box
Circularity
Spatial moments
B. Leibe
53
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Area and Centroid
• We denote the set of pixels in a region by R
• Assuming square pixels, we obtain

Area:
1
A
( x , y )R

Centroid:
x
y
Source: Shapiro & Stockman
1
A
1
A
x
( x , y )R
y
( x , y )R
B. Leibe
54
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Circularity
• Measure the deviation from a perfect circle
R
( x, y )
Circularity:
C
R

where  R and  R2 are the mean and variance of the distance from the centroid of
the shape to the boundary pixels (xk,yk).

Mean radial distance:
R 

1
K
K 1
 ( x , y )  ( x, y)
k
k 0
k
Variance of radial distance:
 
2
R
Source: Shapiro & Stockman
K 1
1
K
  (x , y )  (x, y)   
k 0
2
k
B. Leibe
k
R
55
Invariant Descriptors
• Often, we want features independent of
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
location, orientation, scale.
a1, a2 , a3 ,
Slide credit: Kristen Grauman
b1, b2 , b3 ,
B. Leibe
Feature space
distance
56
Central Moments
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
• S is a subset of pixels (region).
• Central (j,k)th moment defined as:
 jk 
j
k
(
x

x
)
(
y

y
)

( x , y )S
• Invariant to translation of S.
• Interpretation:




0th central moment:
2nd central moment:
3rd central moment:
4th central moment:
Slide credit: Kristen Grauman
area
variance
skewness
kurtosis
B. Leibe
57
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Moment Invariants (“Hu Moments”)
• Normalized central moments
 pq
pq
 pq   ,

1
00
2
• From those, a set of invariant moments can be defined
for object description.
1  20  02
2  ( 20   02 ) 2  4112
3  (30  312 ) 2  (3 21  03 ) 2
4  (30  12 ) 2  ( 21  03 ) 2
• Robust to translation, rotation & scaling,
but don’t expect wonders (still summary statistics).
B. Leibe
58
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Moment Invariants
5  (30  312 )(30  12 )(30  12 ) 2  3(21  03 ) 2 

 (321 03 )(21  03 ) 3(30  12 ) 2  (21  03 ) 2
)(
6  (20 02 30  12 )  (21  03 )
 411 (30  12 )(21  03 )
2
2


7  (321 03 )(30  12 )(30  12 ) 2  3(21  03 ) 2 

 (312 30 )(21  03 ) 3(30  12 ) 2  (21  03 ) 2

Often better to use log10(Ái) instead of Ái directly…
B. Leibe
59
Axis of Least Second Moment
• Invariance to orientation?
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer

Need a common alignment
Axis for which the
squared distance to 2D
object points is minimized
(maximized).

Compute Eigenvectors of 2nd moment matrix (Matlab: eig(A))
  20

 11
11 
 v11 v12  1 0   v11 v12 
T
 VDV  

  0   v

02 
v
v
v
2   21
22 
 22 22  
B. Leibe
T
60
Summary: Binary Image Processing
• Pros
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer



Fast to compute, easy to store
Simple processing techniques
Can be very useful for constrained scenarios
• Cons




Hard to get “clean” silhouettes
Noise is common in realistic scenarios
Can be too coarse a representation
Cannot deal with 3D changes
Slide credit: Kristen Grauman
B. Leibe
61
References and Further Reading
• More on morphological operators can be found in
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer

R.C. Gonzales, R.E. Woods,
Digital Image Processing.
Prentice Hall, 2001
• Online tutorial and Java demos available on

http://homepages.inf.ed.ac.uk/rbf/HIPR2/
B. Leibe
62
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Questions ?
B. Leibe
63
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Demo “Haribo Classification”
B. Leibe
64
You Can Do It At Home…
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Accessing a webcam in Matlab:
function out = webcam
% uses "Image Acquisition Toolbox„
adaptorName = 'winvideo';
vidFormat = 'I420_320x240';
vidObj1= videoinput(adaptorName, 1, vidFormat);
set(vidObj1, 'ReturnedColorSpace', 'rgb');
set(vidObj1, 'FramesPerTrigger', 1);
out = vidObj1 ;
cam = webcam();
img=getsnapshot(cam);
B. Leibe
65
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Matlab Intro: Everything is a matrix
Tutorial adapted from W. Freeman, MIT
66
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Matlab Intro: Matrix Index
Tutorial adapted from W. Freeman, MIT
67
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Matlab Intro: Manipulating Matrices
Tutorial adapted from W. Freeman, MIT
68
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Matlab Intro: Manipulating Matrices
Tutorial adapted from W. Freeman, MIT
B. Leibe
69
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Matlab Intro: Scripts and Functions
• Scripts are m-files containing MATLAB statements
• Functions are like any other m-file, but they accept
arguments
• Name the function file the same as the function name
Tutorial adapted from W. Freeman, MIT
B. Leibe
70
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Matlab Intro: Try to Code in Matrix Ways
B. Leibe
71
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Matlab Intro: Important Commands
•
•
•
•
•
•
•
whos
 List variables in workspace
help
 Get help for any command
lookfor
 Search for keywords
clear/clear x  Erase a variable/all variables
save
 Save the workspace
load
 Load a saved workspace
keyboard
 Enter debugging mode (until dbquit)
B. Leibe
72
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Morphology in Matlab
•
•
•
•
•
•
•
N = hist(Y,M)
IM2 = imerode(IM,SE);
IM2 = imdilate(IM,SE);
IM2 = imclose(IM, SE);
IM2 = imopen(IM, SE);
L = bwlabel(BW,N);
STATS = regionprops(L,PROPERTIES) ;




'Area'
'Centroid'
'BoundingBox'
'Orientation‘, …
Slide credit: Kristen Grauman
Augmented Computing
and Sensory
PerceptualVision
WS 12/13
Computer
Questions ?
B. Leibe
74