Transcript 2D Recognition Using SIFT Computer Vision: CS 4731 Computer Science Columbia University A Little Quiz How would you recognize the following types of objects?

2D Recognition Using SIFT
Computer Vision: CS 4731
Computer Science
Columbia University
A Little Quiz
How would you recognize the following types of objects?
Objects on an Assembly Line
License Plates
Rich 2D Images
2D Recognition Using SIFT
Recognize 2D objects in real-world cluttered scenes
using the Scale Invariant Feature Transform (SIFT).
Topics:
(1) Local Appearance & Interest Points
(2) Scale-Space
(3) The SIFT interest point detector
(4) The SIFT descriptor
(5) Matching & Results
Raw Images are Hard to Match
[Different Size, Orientation, Lighting, Color, …]
Removing Sources of Variation
Some Patches are not “Interesting”
Are Lines Interesting?
Are Blobs Interesting and Localizable?
Locating 1D Blobs
Image:
x
I (x )
First Derivative:
I ( x)
x
x
Second Derivative:
 I ( x)
2
x
x
2
Locating 2D Blobs
Use the Laplacian instead of second derivative:
 I ( x, y ) 
2

2
x
2


2
y
2
Location of blobs given by extrema of Laplacian
What about the size?
What is the Size of 2D Blobs?
Scale-Space
Images only store the location of pixels:
I ( x, y )
Scale-Space Images store the “scale” as well:
S ( x, y ,  )
“Scale”
What does “Scale” Mean?
Different image resolutions?
A Continuous Scale-Space Function
We want images like this:
Increasing σ
The Gaussian function can generate these:
G ( x, y ,  ) 
1
2
2
e
 x2  y2

 2 2





S ( x, y ,  )  G ( x, y ,  ) * I ( x, y )
1D Scale-Space Extrema
We want to find all locations x* and scales σ* of S(x,σ)
S ( x,  )  G ( ) * I ( x )
 G ( ) * I ( x )
2
[Lindeberg, 1994]
1D Scale-Space Extrema
We want to find all locations x* and scales σ* of S(x,σ)
S ( x,  )  G ( ) * I ( x )
 G ( ) * I ( x )
2
All blue lines are at x* (by definition)
All yellow dots are at σ*
Whatever the image scale, we can always find all extrema at greater scales
[Lindeberg, 1994]
Characteristic Scales for 2D Images
2
σ1
1
σ2
σ
σ
  G ( ) * I ( x, y )
2
2
[Mikolajczyk, 2002]
Summary of Theory
Interesting Locations:
( x*, y*)  arg max  I ( x, y )
2
( x, y )
Characteristic Scales:
 *  arg max  G ( ) * I ( x, y )
2

The SIFT Detector
An efficient Difference of Gaussians detector
Extracting SIFT Interest Points
Image
Scale-Space Images
Detected Points
*G(σ1)
LoG(σ1)
S(σ2)
LoG(σ2)
*G(σ2)
S(σ3)
…
*G(σ3)
…
I(x,y)



extrema
S(σ1)
filter bad points
(weak extrema,
asymmetric,
bad contrast)
[Lowe, 2004; Brown and Lowe, 2002]
SIFT Detection Examples
The SIFT Descriptor
“Describe” points so they can be compared
SIFT Scale Invariance
Use the characteristic scales to match sizes
Computing the Principal Orientation
Use the histogram of gradient directions
Principal Orientation
SIFT Rotation Invariance
Use the principal orientations to match rotation
Computing the SIFT Descriptor
Histograms of gradient directions over spatial regions
SIFT Lighting Invariance
Affine lighting model
Image Irradiance
Ideal
Offset
Gain
Gradients remove offset
Scene Radiance
Descriptor
Normalization removes gain
SIFT Descriptor Comparison DEMO
SIFT Scale Invariance Results
SIFT Rotation Invariance Results
SIFT Lighting Invariance Results
SIFT Robustness to Clutter
Matching SIFT Descriptors
Object
Scene
Probabilities
SIFT Object Matcher DEMO
SIFT for 3D Objects?
References
[Autopano] Software to make panaromas using SIFT. http://user.cs.tuberlin.de/~nowozin/autopano-sift/
[Brown and Lowe, 2002] M. Brown and D. Lowe. “Invariant Features from
Interest Point Groups”. BMVC, 2002.
[Harris and Stephens, 1988] C. Harris and M. Stephens. “A Combined
Corner and Edge Detector”. 4th Alvey Vision Conference, 1988.
[Lowe, 2004] D. Lowe.“Distinctive Image Features from Scale-Invariant
Keypoints”. IJCV, 2004.
[Lindeberg, 1994] T. Lindeberg. “Scale-Space Theory: A Basic Tool for
Analysing Structures at Different Scales.” J. of Applied Statistics, 1994.
[Matas et al., 2002] J. Matas, O. Chum, M. Urban, and T. Pajdla. “Robust
Wide Baseline Stereo from Maximally Stable Extremal Regions. BMVC, 2002.
[Mikolajczyk, 2002] K. Mikolajczyk. “Detection of Local Features Invariant
to Affine Transformations.” Ph.D. Thesis, 2002.
References
[Mikolajczyk and Schmid, 2004] K. Mikolajczyk and C. Schmid. “Scale
and Affine Invariant Interest Point Detectors.” IJCV, 2004.
[Mikolajczyk and Schmid, 2005] K. Mikolajczyk and C. Schmid. “A
Performance Evaluation of Local Descriptors.” PAMI, 2005.
[SIFT] SIFT Binaries. http://www.cs.ubc.ca/~lowe/keypoints/
[Witkin, 1983] A. Witkin. “Scale-Space Filtering”. IJCAI, 1983.