Transcript Causes of colour

EECS 274 Computer Vision
Linear Filters and Edges
Linear filters and edges
•
•
•
•
Linear filters
Scale space
Gaussian pyramid and wavelets
Edges
• Reading: Chapters 4 and 5 of FP, Chapter
3 of S
Linear filters
• General process:
– Form new image whose pixels
are a weighted sum of original
pixel values, using the same
set of weights at each point
• Properties
– Output is a linear function of
the input
– Output is a shift-invariant
function of the input (i.e. shift
the input image two pixels to
the left, the output is shifted
two pixels to the left)
1
R ij 
( 2 k  1) 2
• Example: smoothing by
averaging
– form the average of pixels in a
neighborhood
• Example: smoothing with a
Gaussian
– form a weighted average of
pixels in a neighborhood
• Example: finding a derivative
– form a weighted average of
pixels in a neighborhood
u i k v  j  k

 Fuv 
u i k v  j  k
1
( 2 k  1) 2
F
uv
u ,v
Convolution
• Represent these weights as an
image, h
• h is usually called the kernel
• Operation is called
convolution
– it’s associative
g ( i , j )   f (i  k , j  l ) h ( k , l )
k ,l
g ( i , j )   f (i  k , j  l ) h ( k , l )
k ,l
g  f h
g  f *h
• Notation in textbook:
Rij   H i u , j  v Fu ,v
u ,v
R H F
• Notice weird order of indices
– all examples can be put in this
form
– it’s a result of the derivation
expressing any shift-invariant
linear operator as a
convolution
Convolution
g (i , j )   f (i  k , j  l ) h ( k , l )
k ,l
g (i , j )   f (i  k , j  l ) h ( k , l )
k ,l
Smoothing by averaging
Smoothing with a Gaussian
• Smoothing with an average
actually doesn’t compare at all
well with a defocussed lens
– Most obvious difference is that
a single point of light viewed in
a defocussed lens looks like a
fuzzy blob; but the averaging
process would give a little
square
– Narrow vertical and horizontal
bards known as ringing effects
• A Gaussian gives a good model
of a fuzzy blob
An isotropic Gaussian
• The picture shows a
smoothing kernel proportional
to
  x2  y2
1
G ( x , y ) 
exp   
2
2
2 
2

 

 


(which is a reasonable model of a
circularly symmetric fuzzy
blob)
Smoothing with a Gaussian
Differentiation and convolution
• Recall
f
f ( x   , y )  f ( x, y )
 lim
x   0

• Now this is linear and shift
invariant, so must be the result
of a convolution
0 0 0 


H   1 0  1
0 0 0 


• We could approximate this as
f ( x n 1 , y )  f ( x n , y )
f

x
x
(which is obviously a convolution;
it’s not a very good way to do
things, as we shall see)
Finite differences
Partial derivative in y axis,
respond strongly to
horizontal edges
Partial derivative in x axis,
respond strongly to
vertical edges
Spatial filter
• Different approximate operators
 f 
2 1/ 2
2
 
 f   f  

x
f   f , | f |      
 
 x   y  
 y 
 
f  [( z5  z8 ) 2  ( z5  z6 ) 2 ]1/ 2
M ( x, y )  G x  G y
f | ( z5  z8 ) |  | z5  z6 ) |
f
 ( z
 z 9 )  ( z 6  z8 )
2
5

2 1/ 2
f | ( z5  z9 ) |  | z6  z8 ) |
z1
z2
z3
z4
z5
z6
z7
z8
z9
Roberts operator
One of the earliest edge detection algorithm by Lawrence Roberts
Two kernels:
Symmetric, tuned to diagonal
Sobel operator
One of the earliest edge detection algorithm by Irwin Sobel
Noise
• Simplest noise model
– independent stationary
additive Gaussian noise
– the noise value at each pixel is
given by an independent draw
from the same normal
probability distribution
• Issues
– this model allows noise values
that could be greater than
maximum camera output or
less than zero
– for small standard deviations,
this isn’t too much of a
problem - it’s a fairly good
model
– independence may not be
justified (e.g. damage to lens)
– may not be stationary (e.g.
thermal gradients in the ccd)
Additive Gaussian noise
sigma=1
sigma=16
Finite differences and noise
• Finite difference filters respond
strongly to noise
– obvious reason: image noise
results in pixels that look very
different from their neighbors
• Generally, the larger the noise
the stronger the response
• What is to be done?
– intuitively, most pixels in
images look quite a lot like
their neighbors
– this is true even at an edge;
along the edge they’re similar,
across the edge they’re not
– suggests that smoothing the
image should help, by forcing
pixels different to their
neighbors (=noise pixels?) to
look more like neighbors
Finite differences responding to
noise
σ=0.03
σ=0.09
Increasing noise -> (this is zero mean additive Gaussian noise)
Difference operation is strongly influenced by noise (the image is
increasingly grainy)
Smoothed noise
Smoothing stationary additive Gaussian noise results in signals
where pixel values tend to increasingly similar to the value of
neighboring pixels (as filter kernel causes correlation)
Smoothing reduces noise
• Generally expect pixels to “be
like” their neighbors
– surfaces turn slowly
– relatively few reflectance
changes
• Generally expect noise
processes to be independent
from pixel to pixel
• Implies that smoothing
suppresses noise, for
appropriate noise models
• Scale
– the parameter in the
symmetric Gaussian
– as this parameter goes up,
more pixels are involved in the
average
– and the image gets more
blurred
– and noise is more effectively
suppressed
Effects of smoothing
The effects of smoothing
• Each row shows smoothing
with Gaussians of different
width
• Each column shows
images with different
Gaussian noise
Gradients and edges
• Points of sharp change in an
image are interesting:
–
–
–
–
change in reflectance
change in object
change in illumination
noise
• Sometimes called edge points
• General strategy
– determine image gradient
– now mark points where
gradient magnitude is
particularly large wrt neighbors
(ideally, curves of such points)
In one dimension, the 2nd derivative of a signal is zero when the
derivative magnitude is extremal  a good place to look for edge
is where the second derivative is zero
Smoothing and differentiation
• Issue: noise
– smooth before differentiation
– two convolutions to smooth, then differentiate?
– actually, no - we can use a derivative of Gaussian
filter
• because differentiation is convolution, and convolution is
associative
Smoothed derivative of
Gaussian filter in the
x-direction
Smoothed derivative of
Gaussian filter in the
y-direction
Scale of smoothing
1 pixel
3 pixels
7 pixels
The scale of the smoothing filter affects derivative estimates, and also
the semantics of the edges recovered
Image gradient
• Gradient equation:
• Represents direction of most rapid change in intensity
• Gradient direction:
• The edge strength is given by the gradient magnitude
Theory of edge detection
Ideal edge
L  x , y   x sin   y cos     0
B1 : L  x , y   0
B2 : L  x , y   0
Unit step function:
1

u t    1
2
0

for t  0
for t  0
for t  0
u t  
  s ds
t

Image intensity (brightness):
I  x , y   B 1   B 2  B 1 u  x sin   y c o s    
Theory of edge detection
• Image intensity (brightness):
I  x , y   B 1   B 2  B 1 u  x sin   y c o s    
• Partial derivatives (gradients):
I
  sin   B2  B1   x sin   y cos    
x
I
  cos   B2  B1   x sin   y cos    
y
• Squared gradient:
2
 I   I 
2
s  x , y          B 2  B1   x sin   y cos    
 x   y 
Edge Magnitude:
sx, y 
 I I 
 /  (normal of the edge)
arctan
Edge Orientation:
 y x 
2
Rotationally symmetric, non-linear operator
Theory of edge detection
• Image intensity (brightness):
I  x , y   B 1   B 2  B 1 u  x sin   y c o s    
• Partial derivatives (gradients):
I
  sin   B2  B1   x sin   y cos    
x
I
  cos   B2  B1   x sin   y cos    
y
• Laplacian:
2I 2I
 I  2  2   B 2  B1  '  x sin   y cos    
x
y
2
I
x
Rotationally symmetric, linear operator
2 I
x2
zero-crossing
Discrete edge operators
• How can we differentiate a discrete image?
Finite difference approximations:
I
1
I i 1, j 1  I i , j 1   I i 1, j  I i , j 

x 2
I
1
I i 1, j 1  I i 1, j   I i , j 1  I i , j 

y 2
Convolution masks :
I
1

 x 2
I
1

y 2
I i , j 1 Ii 1, j 1
Ii, j
I i 1, j
Discrete edge operators
• Second order partial derivatives:
Ii 1, j 1 I i , j 1 Ii 1, j 1
2I
1
I i 1, j  2 I i , j  I i 1, j 

2
2
x

2I
1
I i , j 1  2 I i , j  I i , j 1 

2
2
y

I i 1, j I i , j I i 1, j
Ii 1, j 1 I i , j 1 Ii 1, j 1
• Laplacian :
2
2

I

I
2I  2  2
x
y
Convolution masks :
2I 
1
2
0
1
0
1
-4
1
0
1
0
or
1
6 2
1
4
1
4
-20
4
1
4
1
(more accurate)
Effects of noise
• Consider a single row or column of the image
– Plotting intensity as a function of position gives a signal
Where is the edge?
Solution: Smooth first
Where is the edge?
Look for peaks in
Derivative theorem of
convolution
…saves us one operation
Laplacian of Gaussian (LoG)
By Marr and Hildreth
 2
2
h  f    2
2
x
 x

h   f

Laplacian of Gaussian
Laplacian of Gaussian
operator
Where is the edge?
Zero-crossings of bottom graph !
2D Gaussian edge operators
Gaussian
Derivative of Gaussian (DoG)
Laplacian of Gaussian
Mexican Hat (Sombrero)
•
is the Laplacian operator:
Can be substituted with two DoGs
threshold
sigma=2
scale
contrast=1
LOG zero crossings
contrast=4
sigma=4
How to find best scale and threshold?
Ringing effects
LoG is not oriented (isotropic)
We still have unfortunate behavior at corners and trihedral areas
σ=1 pixel
σ=2 pixel
From thick trails to curves (formed by points)
Three major issues:
1) The gradient magnitude at different scales is different; which should
we choose?
2) The gradient magnitude is large along thick trail; how
do we identify the significant points?
3) How do we link the relevant points up into curves?
Gradient-based edge detectors
Trail of large gradient
magnitude
Appropriate cutting
direction
Peak in this direction
• Wish to mark points along the curve where the magnitude is biggest
• Looking for a maximum along a slice normal to the curve (non-maximum
suppression)
• These points should form a curve
• Two algorithmic issues:
• at which point is the maximum
• where is the next one?
Non-maximum suppression
• Check if pixel is local maximum along gradient direction
– requires checking interpolated pixels p and r
Edge following
t
Predicting
the next
edge point
Assume the marked point is an edge point. Then we construct
the tangent to the edge curve (which is normal to the gradient
at that point) and use this to predict the next points (here either
s or t)
Edge following: edge points occur along curve like chains
Edge points
coarse scale, high threshold σ=4 pixels
fine scale, high threshold σ=1 pixel
coarse scale, low threshold σ=4 pixels
Canny edge operator
• Smooth image I with 2D Gaussian:
GI
• Find local edge normal directions for each pixel
 G  I 
n
 G  I 
• Compute edge magnitudes
 G  I 
• Locate edges by finding zero-crossings along the edge normal
directions (non-maximum suppression)
 2 G  I 
0
2
n
Canny edge detector
Original image
Magnitude of gradient
Canny edge detector
After non-maximum suppression
Canny edge detector
original
Canny with
• The choice of
– large
– small
Canny with
depends on desired behavior
detects large scale edges
detects fine features
Difference of Gaussians (DoG)
• Laplacian of Gaussian can be approximated by the
difference between two different Gaussians
DoG edge detection
(a)
(b)
(b)-(a)
Unsharp masking
100
200
–
300
=
400
500
200
400
+a
blurred positive
600
increase details in bright area
800
=
Edge thresholding
• Standard Thresholding:
• Can only select “strong” edges.
• Does not guarantee “continuity” (i.e., lead to broken edge curves)
• Hysteresis based Thresholding (use two thresholds)
Example: For “maybe” edges, decide on the edge if neighboring pixel is a strong edge
Remaining issues
• Check that maximum value of gradient
value is sufficiently large
– drop-outs? use hysteresis
• use a high threshold to start edge curves and a low
threshold to continue them
Notes
• Something nasty is happening at corners
• Scale affects contrast
• Edges are not bounding contours
Scale space
• Framework for multi-scale
signal processing
• Define image structure in
terms of scale with kernel
size, σ
• Find scale invariant
operations
• See Scale-Space Theory
in Computer Vision by
Tony Lindeberg
Orientation representations
• The gradient magnitude is
affected by illumination
changes
– but it’s direction isn’t
• We can describe image
patches by the swing of the
gradient orientation
• A variety of specialized corner
detector
• Important types:
– constant window
• small gradient magnitudes
– edge window
• few large gradient
magnitudes in one
direction
– flow window
• many large gradient
magnitudes in one
direction
– corner window
• large gradient magnitudes
that change
Around the corners and in the textured regions, the orientation vector swings sharply
Representing windows
Looking at variations
H
 (I )(I )
T
window
  G

  G
  G
  G

 I    I     I    I  
  x
  x
  x
  y

  
  G
 G

 G
window  G
 
 I 
 I  
 I 
 I  
  y
  y
  y

  x
•
Types
– constant
• small eigenvalues
– edge
• one medium, one small
– flowf
• one large, one small
– corner
• two large eigenvalues
Plotting in ellipses to understand the matrix
(variation of gradient) of 3 × 3 window
Orientation field
( x, y )T H
1
(x, y)  
Major and minor axes are along
the eigenvectors of H, and the extent
corresponds to the size of eigenvalues
Orientation field
Plotting in ellipses to understand
the matrix (variation of gradient)
of 5 × 5 window
Corners
• Harris corner detector
• Moravec corner detector
• SIFT descriptors
(more in the next lectures)
Filters are templates
• Applying a filter at some point
can be seen as taking a dotproduct between the image
and some vector
• Filtering the image is a set of
dot products
convolution is equivalent to
taking the dot product of the filter
with an image patch
• Insight
Rij   H i u , j  v Fu ,v
u ,v
Rij   H u ,  v Fu ,v
u ,v
– filters look like the effects they
are intended to find
– filters find effects they look like
derivative of Gaussian used as edge
detection
Normalized correlation
• Think of filters of a dot product
– now measure the angle
– i.e normalized correlation
output is filter output, divided
by root sum of squares of
values over which filter lies
– cheap and efficient method for
finding patterns
• Tricks:
– ensure that filter has a zero
response to a constant region
(helps reduce response to
irrelevant background)
– subtract image average when
computing the normalizing
constant (i.e. subtract the
image mean in the
neighborhood)
– absolute value deals with
contrast reversal
Finding hands
Controlling TV set based on normalized correlation score of patterns by Freeman et al, 98
Anisotropic scaling
• Symmetric Gaussian smoothing tends to blur out
edges rather aggressively
• Prefer an oriented smoothing operator that
smoothes
– aggressively perpendicular to the gradient
– little along the gradient
• Also known as edge preserving smoothing
• Formulated with diffusion equation
Diffusion equation for
anisotropic filter
• PDE that describes fluctuations in a material
undergoing diffusion



 (r , t )
    D ( , r )   ( r , t ) 
t
• For isotropic filter

 2  2
2
   ( c ( x , y ,  ))   )  C    2  2   2 for istropic cases

x
y
 ( x , y ,0 )  I ( x , y ) as initial condition
• For anisotropic filter

   ( c ( x , y ,  ))   )  c ( x , y ,  ) 2  ( c ( x , y ,  ))   

If c ( x , y ,  )  1, just as before
If c ( x , y ,  )  0, no smoothing
Anisotropic filtering
Often used in graphics for textured surface to reduce aliasing effects
Anisotropic diffusion
Input image with noise
Linear isotropic diffusion (simple Gaussian)
Nonlinear anisotropic diffusion
Nonlinear isotropic diffusion (edge skipping)
Perona and Malik 90
Bilateral filtering
• Preserving edges
– Replace pixel’s value by a weighted average
of its neighbors in both space and intensity
– Controlled by range and spatial parameters
One iteration
Tomasi and Manduchi 98
Multiple iterations