Transcript Document

Harris Corner Detector
&
Scale Invariant Feature Transform
(SIFT)
Harris Corner Detector
Harris Detector: Intuition
“flat” region:
no change in all
directions
“edge”:
no change along
the edge direction
“corner”:
significant change
in all directions
Moravec Corner Detector
• Shift in any direction would result in a significant change
at a corner.
Algorithm:
•Shift in horizontal,
vertical, and diagonal
directions by one pixel.
•Calculate the absolute
value of the MSE for each
shift.
•Take the minimum as the
cornerness response.
Harris Detector: Mathematics
Change of intensity for the shift [u,v]:
E (u , v)   w( x, y )  I ( x  u , y  v)  I ( x, y ) 
2
x, y
Window
function
Shifted
intensity
Window function w(x,y) =
Intensity
or
1 in window, 0 outside
Gaussian
Harris Detector: Mathematics
Apply Taylor series expansion:
E (u , v)  Au 2  2Cuv  Bv 2
A   w( x, y ) I x2 ( x, y )
x, y
B   w( x, y ) I y2 ( x, y )
x, y
C   w( x, y ) I x ( x, y ) I y ( x, y )
x, y
 A C  u 
E (u, v)  u v  
 v 
C
B

 
Harris Detector: Mathematics
For small shifts [u,v] we have the following approximation:
E (u, v)  u, v 
u 
M  
v 
where M is a 22 matrix computed from image derivatives:
 I x2
M   w( x, y ) 
x, y
 I x I y
IxI y 
2 
I y 
Harris Detector: Mathematics
Intensity change in shifting window: eigenvalue analysis
E (u, v)  u, v 
u 
M  
v 
1, 2 – eigenvalues of M
direction of the
fastest change
Ellipse E(u,v) = const
direction of the
slowest change
(max)-1/2
(min)-1/2
Harris Detector: Mathematics
Classification of
image points using
eigenvalues of M:
2
“Edge”
2 >> 1
“Corner”
1 and 2 are large,
1 ~ 2 ;
E increases in all
directions
1 and 2 are small;
E is almost constant
in all directions
“Flat”
region
“Edge”
1 >> 2
1
Harris corner detector
Measure of corner response:
R  det M  k  trace M 
2
det M  12
trace M  1  2
(k – empirical constant, k = 0.04-0.06)
No need to compute eigenvalues explicitly!
Eliminate small responses.
Find local maxima of the remaining.
Harris Detector: Scale
Rmin= 0
Rmin= 1500
Summary of the Harris detector
Harris Detector: Some
Properties
• Rotation invariance
Ellipse rotates but its shape (i.e. eigenvalues) remains
the same
Corner response R is invariant to image rotation
Harris Detector: Some
Properties
• Partial invariance to affine intensity change
 Only derivatives are used => invariance to
intensity shift I  I + b
 Intensity scale: I  a I
R
R
threshold
x (image coordinate)
x (image coordinate)
Harris Detector: Some
Properties
• But: non-invariant to image scale!
All points will be
classified as edges
Corner !
Harris Detector: Some
Properties
• Quality of Harris detector for different
scale changes
Repeatability rate:
# correspondences
# possible correspondences
C.Schmid et.al. “Evaluation of Interest Point Detectors”. IJCV 2000
Scale Invariant Detection
• Consider regions (e.g. circles) of different
sizes around a point
• Regions of corresponding sizes will look the
same in both images
Scale Invariant Detection
• The problem: how do we choose
corresponding circles independently in each
image?
Scale Invariant Detection
• Solution:
– Design a function on the region (circle), which is
“scale invariant” (the same for corresponding
regions, even if they are at different scales)
Example: average intensity. For corresponding
regions (even of different sizes) it will be the same.
– For a point in one image, we can consider it as a function of
region size (circle radius)
f
Image 1
f
Image 2
scale = 1/2
region size
region size
Scale Invariant Detection
• Common approach:
Take a local maximum of this function
Observation: region size, for which the maximum is achieved,
should be invariant to image scale.
Image 1
f
f
Image 2
scale = 1/2
s1
region size
s2
region size
Characteristic Scale
Ratio of scales corresponds to a scale factor between two images
Scale Invariant Detection
• A “good” function for scale detection:
has one stable sharp peak
f
f
bad
region size
f
Good !
bad
region size
region size
• For usual images: a good function would be a
one which responds to contrast (sharp local
intensity change)
Scale Invariant Detection
• Functions for determining scale
f  Kernel  Image
Kernels:
L   2  Gxx ( x, y,  )  G yy ( x, y,  ) 
(Laplacian)
DoG  G( x, y, k )  G( x, y,  )
(Difference of Gaussians)
where Gaussian
G ( x, y ,  ) 
1
2

e
x y
2
2 2
2
L or DoG kernel is a matching filter.
It finds blob-like structure. It turns
out to be also successful in getting
characteristic scale of other
structures, such as corner regions.
Difference-of-Gaussians
 
G k 2 * I
Gk * I
G * I
D   Gk   G * I
Scale-Space Extrema
• Choose all extrema within 3x3x3 neighborhood.
 
D k 2
Dk 
D  
X is selected if it is larger or smaller than all 26 neighbors
Scale Invariant Detectors
Find local maximum of:
– Harris corner detector in
space (image coordinates)
– Laplacian in scale
• SIFT (Lowe)2
Find local maximum of:
– Difference of Gaussians in
space and scale
1 K.Mikolajczyk,
 Laplacian 
scale
y
 Harris 
x
 DoG 
x
scale
 DoG 
•
Harris-Laplacian1
y
C.Schmid. “Indexing Based on Scale Invariant Interest Points”. ICCV 200
2 D.Lowe. “Distinctive Image Features from Scale-Invariant Keypoints”. Accepted to IJCV
Harris-Laplace Detector
Scale Invariant Detectors
• Experimental evaluation of detectors
w.r.t. scale change
Repeatability rate:
# correspondences
# possible correspondences
K.Mikolajczyk, C.Schmid. “Indexing Based on Scale Invariant Interest Points”. ICCV 2001
Affine Invariant Detection
• Above we considered:
Similarity transform (rotation + uniform scale)
• Now we go on to:
Affine transform (rotation + non-uniform
scale)
Affine Invariant Detection
• Take a local intensity extremum as initial point
• Go along every ray starting from this point and stop
when extremum of function f is reached
f
I (t )  I 0
f (t ) 
t
1
points along the
ray
t
 I (t )  I
0
dt
o
T.Tuytelaars, L.V.Gool. “Wide Baseline Stereo Matching Based on Local,
Affinely Invariant Regions”. BMVC 2000.
Affine Invariant Detection
• Such extrema occur at
positions where intensity
suddenly changes compared to
the intensity changes up to that
point.
• In theory, leaving out the
denominator would still give
invariant positions. In practice,
the local extrema would be
shallow, and might result in
inaccurate positions.
I (t )  I 0
f (t ) 
t
1
t
 I (t )  I
o
T.Tuytelaars, L.V.Gool. “Wide Baseline Stereo Matching Based on Local,
Affinely Invariant Regions”. BMVC 2000.
0
dt
Affine Invariant Detection
• The regions found may not exactly correspond, so we
approximate them with ellipses
• Find the ellipse that best fits the region
Affine Invariant Detection
•
Covariance matrix of region points defines an ellipse:
q  Ap
1
1
p  p 1
T
1  ppT
qT 21q  1
 2  qqT
region 1
( p = [x, y]T is relative to
the center of mass)
2  A1 AT
Ellipses, computed for corresponding regions,
also correspond!
region 2
Affine Invariant Detection
• Algorithm summary (detection of affine invariant region):
– Start from a local intensity extremum point
– Go in every direction until the point of extremum of
some function f
– Curve connecting the points is the region boundary
– Compute the covariance matrix
– Replace the region with ellipse
T.Tuytelaars, L.V.Gool. “Wide Baseline Stereo Matching Based on Local,
Affinely Invariant Regions”. BMVC 2000.
Affine Invariant Detection
• Maximally Stable Extremal
Regions
– Threshold image intensities: I > I0
– Extract connected components
(“Extremal Regions”)
– Find “Maximally Stable” regions
– Approximate a region with
an ellipse
J.Matas et.al. “Distinguished Regions for Wide-baseline Stereo”. Research Report of
Affine Invariant Detection :
Summary
• Under affine transformation, we do not know in
advance shapes of the corresponding regions
• Ellipse given by geometric covariance matrix of a
region robustly approximates this region
• For corresponding regions ellipses also correspond
Methods:
1. Search for extremum along rays [Tuytelaars, Van Gool]:
2. Maximally Stable Extremal Regions [Matas et.al.]
Point Descriptors
• We know how to detect points
• Next question:
How to match them?
?
Point descriptor should be:
1. Invariant
2. Distinctive
Descriptors Invariant to
Rotation
• Convert from Cartesian to Polar coordinates
• Rotation becomes translation in polar
coordinates
• Take Fourier Transform
– Magnitude of the Fourier transform is invariant to
translation.
Descriptors Invariant to
Rotation
• Find local orientation
Dominant direction of gradient
• Compute image regions relative to this
orientation
Descriptors Invariant to Scale
• Use the characteristic scale determined by
detector to compute descriptor in a normalized
frame
Affine Invariant Descriptors
• Find affine normalized frame
A
 2  qqT
1  ppT
11  A1T A1
A1
A2
21  A2T A2
rotation
• Compute rotational invariant descriptor in this
normalized frame
J.Matas et.al. “Rotational Invariants for Wide-baseline Stereo”. Research Report of CMP,
Affine covariant regions
SIFT
(Scale Invariant Feature Transform)
SIFT – Scale Invariant Feature
Transform1
• Empirically found2 to show very good performance,
invariant to image rotation, scale, intensity change,
and to moderate affine transformations
Scale = 2.5
Rotation = 450
1 D.Lowe.
“Distinctive Image Features from Scale-Invariant Keypoints”. Accepted to IJCV
2 K.Mikolajczyk, C.Schmid. “A Performance Evaluation of Local Descriptors”. CVPR 2003
SIFT – Scale Invariant Feature
Transform
• Descriptor overview:
– Determine scale (by maximizing DoG in scale and in space),
local orientation as the dominant gradient direction.
Use this scale and orientation to make all further
computations invariant to scale and rotation.
– Compute gradient orientation histograms of several small
windows (to produce 128 values for each point)
D.Lowe. “Distinctive Image Features from Scale-Invariant Keypoints”. Accepted to IJCV 2
Scale-space extrema detection
• Need to find “characteristic scale” for each
feature point.
Difference-of-Gaussians
 
G k 2 * I
Gk * I
G * I
D   Gk   G * I
Scale-Space Extrema
• Choose all extrema within 3x3x3 neighborhood.
 
D k 2
Dk 
D  
X is selected if it is larger or smaller than all 26 neighbors
Keypoint Localization & Filtering
• Now we have much less points than
pixels.
• However, still lots of points (~1000s)…
– With only pixel-accuracy at best
– And this includes many bad points
Brown & Lowe 2002
Keypoint Filtering - Low
Contrast
• Reject points with bad contrast:
– DoG smaller than 0.03 (image values in [0,1])
• Reject edges
– Similar to the Harris detector; look at the
autocorrelation matrix
Maxima in D
Remove low contrast and edges
Orientation assignment
• By assigning a consistent orientation, the keypoint
descriptor can be orientation invariant.
• Let, for a keypoint, L is the image with the closest scale.
– Compute gradient magnitude and orientation using finite
differences:
 L( x  1, y)  L( x  1, y) 
GradientVector  

L
(
x
,
y

1)

L
(
x
,
y

1)


Orientation assignment
Orientation assignment
Orientation assignment
Orientation assignment
Orientation Assignment
• Any peak within 80% of the highest peak
is used to create a keypoint with that
orientation
• ~15% assigned multiplied orientations, but
contribute significantly to the stability
SIFT descriptor
SIFT Descriptor
• Each point so far has x, y, σ, m, θ
• Now we need a descriptor for the region
– Could sample intensities around point, but…
• Sensitive to lighting changes
• Sensitive to slight errors in x, y, θ
Edelman et al. 1997
SIFT Descriptor
• 16x16 Gradient window is taken. Partitioned into 4x4 subwindows.
• Histogram of 4x4 samples in 8 directions
• Gaussian weighting around center(  is 0.5 times that of the scale of
a keypoint)
• 4x4x8 = 128 dimensional feature vector
Image from: Jonas Hurrelmann
Performance
• Very robust
– 80% Repeatability at:
• 10% image noise
• 45° viewing angle
• 1k-100k keypoints in database
• Best descriptor in [Mikolajczyk &
Schmid 2005]’s extensive survey
Recognition under occlusion