Transcript Linear Filters - University of Delaware

Linear Filters
• General process:
– Form new image whose pixels
are a weighted sum of original
pixel values, using the same set
of weights at each point.
• Properties
– Output is a linear function of
the input
– Output is a shift-invariant
function of the input (i.e. shift
the input image two pixels to
the left, the output is shifted
two pixels to the left)
• Example: smoothing by
averaging
– form the average of pixels in a
neighbourhood
• Example: smoothing with a
Gaussian
– form a weighted average of
pixels in a neighbourhood
• Example: finding a derivative
– form a weighted average of
pixels in a neighbourhood
Computer Vision : CISC 4/689
Convolution
• Represent these weights as an
image, H
• H is usually called the kernel
• Operation is called convolution
• Result is:
– it’s associative
Computer Vision : CISC 4/689
Rij   H iu, jvFuv
u,v
Example: Smoothing by Averaging
Computer Vision : CISC 4/689
Smoothing with a Gaussian
• Smoothing with an average
actually doesn’t compare at all
well with a defocussed lens
– Most obvious difference is that
a single point of light viewed
in a defocussed lens looks like
a fuzzy blob; but the averaging
process would give a little
square.
• A Gaussian gives a good model of
a fuzzy blob
Computer Vision : CISC 4/689
An Isotropic Gaussian
• The picture shows a smoothing
kernel proportional to
 x 2  y 2 
exp  

2
  2 
(which is a reasonable model of a
circularly symmetric fuzzy
blob)
Computer Vision : CISC 4/689
Smoothing with a Gaussian
Computer Vision : CISC 4/689
Problem: Image Noise
from Forsyth & Ponce
Computer Vision : CISC 4/689
Solution: Smoothing (Low-Pass) Filters
•
If object reflectance changes slowly and noise at each pixel is independent, then we
want to replace each pixel with something like the average of neighbors
– Disadvantage: Sharp (high-frequency) features lost
Original image
Computer Vision : CISC 4/689
7 x 7 averaging
neighborhood
Smoothing Filters: Details
• Filter types
– Mean filter (box)
– Median (nonlinear)
– Gaussian
• Can specify linear operation by
shifting kernel over image and
taking product
Computer Vision : CISC 4/689
1
1
1
1
1
1
1
1
1
3 x 3 box filter kernel
Gaussian Kernel
•
Idea: Weight contributions of neighboring pixels by nearness
0.003
0.013
0.022
0.013
0.003
0.013
0.059
0.097
0.059
0.013
0.022
0.097
0.159
0.097
0.022
0.013
0.059
0.097
0.059
0.013
5 x 5,  = 1
•
Smooth roll-off reduces “ringing” seen in box filter
Computer Vision : CISC 4/689
0.003
0.013
0.022
0.013
0.003
Gaussian Smoothing
•
In theory, the Gaussian distribution is non-zero everywhere, which would require an
infinitely large convolution mask, but in practice it is effectively zero more than about
three standard deviations from the mean, and so we can truncate the mask at this point
•
Discrete approximation to Gaussian function with sd = 1.4
Source: http://www.cee.hw.ac.uk/hipr/html/gsmooth.html
Computer Vision : CISC 4/689
Gaussian Smoothing
•
Once a suitable mask has been calculated, then the Gaussian smoothing can be
performed using standard convolution methods. The convolution can in fact be
performed fairly quickly since the equation for the 2-D isotropic Gaussian
shown above is separable into x and y components. Thus the 2-D convolution
can be performed by first convolving with a 1-D Gaussian in the x direction,
and then convolving with another 1-D Gaussian in the y direction. (The
Gaussian is in fact the only completely circularly symmetric operator which
can be decomposed in such a way.) Below shows the 1-D x component mask
that would be used to produce the full mask shown in previous slide. The y
component is exactly the same but is oriented vertically.
A further way to compute a Gaussian smoothing with a large standard
deviation is to convolve an image several times with a smaller Gaussian.
While this is computationally complex, it can have applicability if the
processing is carried out using a hardware pipeline.
Computer Vision : CISC 4/689
Gaussian Smoothing
•
One of the principle justifications for using the Gaussian as a smoothing filter
is due to its frequency response. Most convolution based smoothing filters act
as lowpass frequency filters. This means that their effect is to remove low
spatial frequency components from an image. The frequency response of a
convolution filter, i.e. its effect on different spatial frequencies, can be seen by
taking the Fourier transform of the filter.
•
•
Frequency responses of Box (i.e. mean) filter (width 7 pixels) and Gaussian
filter ( = 3 pixels).
Computer Vision : CISC 4/689
Example: Gaussian Smoothing
Original image
Box filter
7x7
kernel
=1
Computer Vision : CISC 4/689
=3
Convolution
•
•
Formalization of idea of overlap
The animations above graphically illustrate the convolution
(product) of two functions as one is of two rectangle functions (left) and two Gaussians (right).
In the plots, the green curve shows the convolution of the blue
shifted across the other
and red curves as a function of t , the position indicated by
The area under convolution is
the vertical green line. The gray region indicates the product
product of areas under the
as a function of t , so its area is precisely the convolution.
functions
• The convolution of two
Gaussians is another Gaussian
courtesy of mathworld.wolfram.com
Computer Vision : CISC 4/689
Convolution Notes
• Note the assumption that both f
and g are continuous and
defined everywhere
• Properties
– Commutative
– Associative
Computer Vision : CISC 4/689
Discrete 2-D Convolution
•
Sum instead of integral: I’
by:
= K ¤ I (book uses ¤¤ for 2-D convolution) is defined
I’(u, v) = §x, y K(u ¡ x, v ¡ y) I(x, y)
where the kernel K approximates the impulse function f by sampling from it where it
is non-zero
Computer Vision : CISC 4/689
Correlation
• Same as convolution, a dot product, thus it is largest when the pattern
maches (vectors are parallel), this can be used to find texture patterns
• This yields a value that is +ve when the image region looks like the
filter kernel, and small and –ve when it is opposite. Can be squared if
pattern reversal doesn’t matter.
• Since value maybe large if image is bright, so divide by root sum of
squares of image region and filter. (dot of unit vectors)
• Some ways to interpret what the kernel is doing
– As a template being matched by correlation
– As simply a set of weights on the corresponding image pixels
Computer Vision : CISC 4/689
Normalized correlation
• Think of filters of a dot product
– now measure the angle
– i.e normalised correlation
output is filter output, divided
by root sum of squares of
values over which filter lies
• Tricks:
– ensure that filter has a zero
response to a constant region
(helps reduce response to
irrelevant background)
– subtract image average when
computing the normalizing
constant (i.e. subtract the
image mean in the
neighbourhood)
– absolute value deals with
contrast reversal
Computer Vision : CISC 4/689
Positive responses
Zero mean image, -1:1 scale
Zero mean image, -max:max scale
Computer Vision : CISC 4/689
Positive responses
Zero mean image, -1:1 scale
Zero mean image, -max:max scale
Computer Vision : CISC 4/689
Figure from “Computer Vision for Interactive Computer Graphics,” W.Freeman et al, IEEE Computer Graphics and Applications,
1998 copyright 1998, IEEE
Computer Vision : CISC 4/689
Dealing with Image Edges
• Only convolve with interior
– Shrinks image
• Zero-padding
– Results in spurious gradients
• Border replication
• Symmetric: Reflect image at border b so that
I(b + i) = I(b ¡ i)
– Results in spurious 2nd-derivatives
Computer Vision : CISC 4/689
1
1
1
-1
2
1
-1
-1
1
Step 1
1
1
1
-1
42
2
2
3
-1
2
-2
1
3
3
2
2
1
2
1
3
2
2
5
Computer Vision : CISC 4/689
2
2
2
3
2
1
3
3
2
2
1
2
1
3
2
2
1
1
1
-1
2
1
-1
-1
1
Step 2
1
1
1
2
-2
42
2
3
2
-2
1
-1
3
3
2
2
1
2
1
3
2
2
5
Computer Vision : CISC 4/689
4
2
2
2
3
2
1
3
3
2
2
1
2
1
3
2
2
1
1
1
-1
2
1
-1
-1
1
Step 3
1
1
1
2
2
-2
42
3
2
1
-1
3
-3
3
2
2
1
2
1
3
2
2
5
Computer Vision : CISC 4/689
4
2
2
2
3
2
1
3
3
2
2
1
2
1
3
2
2
4
1
1
1
-1
2
1
-1
-1
1
Step 4
1
1
1
2
2
2
-2
36
1
2
1
3
-3
3
-3
1
2
2
1
2
1
3
2
2
5
Computer Vision : CISC 4/689
4
2
2
2
3
2
1
3
3
2
2
1
2
1
3
2
2
4
-2
1
1
1
-1
2
1
-1
-1
1
Step 5
1
2
2
2
3
5
-1
42
1
3
3
9
-1
-2
2
2
1
2
1
3
2
2
Computer Vision : CISC 4/689
4
2
2
2
3
2
1
3
3
2
2
1
2
1
3
2
2
4
-2
1
1
1
-1
2
1
-1
-1
1
Step 6
2
2
2
3
5
4
-2
2
21
3
3
9
6
2
-2
2
-2
1
2
1
3
2
2
Computer Vision : CISC 4/689
2
2
2
3
2
1
3
3
2
2
1
2
1
3
2
2
4
-2
and so on…
Computer Vision : CISC 4/689
1
1
1
-1
2
1
-1
-1
1
Final Result
2
2
2
3
5
4
4
-2
2
1
3
3
9
6
14
5
2
2
1
2
11
7
6
5
1
3
2
2
9
12
8
5
I
I’
Why is I’ large in some places and small in others?
Computer Vision : CISC 4/689
Linear Shift Invariance
• Possible properties of f
– Superposition: f(I1
–
+ I2) = f(I1) + f(I2)
Scaling: f(®I) = ®f(I)
– Shift invariance:
f(Shift(I, k)) = Shift (f(I), k)
A system with these properties is performing convolution
Computer Vision : CISC 4/689
Imaging Systems
• An imaging system describes a functional transformation f of an image due
to…
– Physics: A real-world phenomenon such as blurring from defocus or fish-eye
lens distortion
– Filtering: A transformation we apply in order to
• Undo or mitigate the bad effects of a physical system (e.g., deblur,
undistort, etc.)
• Emphasize or highlight particular image properties (e.g., color similarity,
edges, etc.)
I
I’
f
Computer Vision : CISC 4/689
What’s Not a Convolution?
• Nonlinear systems
– E.g., radial distortion of fish-eye lens is not LSI because geometric
transformation depends on pixel location
f
courtesy of M. Fiala
Computer Vision : CISC 4/689
Filtering in Matlab
• imfilter(I, K)filters image I with kernel K
– Default filtering is correlation (no kernel rotation)
– Can set options on border handling
• corr2, conv2 are the generic versions
• Kernel creation
– Custom (create a matrix)
– fspecial function
Computer Vision : CISC 4/689
Differentiation and convolution
• Recall
• We could approximate this as
f
 f x   , y f x, y 
 lim



0

x

 
• Now this is linear and shift
invariant, so must be the result
of a convolution.
f f xn1 , y  f xn , y

x
x
(which is obviously a convolution;
it’s not a very good way to do
things, as we shall see)
Computer Vision : CISC 4/689
Finite differences
Partial derivative in x-direction: gives vertical stripes.
mid-gray = 0, dark gray = -ve, light gray = +ve
Kernel = (1,0,-1)
Computer Vision : CISC 4/689
Noise
• Simplest noise model
• Issues
– independent stationary additive
Gaussian noise
– the noise value at each pixel is
given by an independent draw
from the same normal
probability distribution
– this model allows noise values
that could be greater than
maximum camera output or
less than zero
– for small standard deviations,
this isn’t too much of a
problem - it’s a fairly good
model
– independence may not be
justified (e.g. damage to lens)
– may not be stationary (e.g.
thermal gradients in the ccd)
Computer Vision : CISC 4/689
sigma=1
Computer Vision : CISC 4/689
sigma=16
Computer Vision : CISC 4/689
Finite differences and noise
• Finite difference filters respond
strongly to noise
– obvious reason: image noise
results in pixels that look very
different from their neighbours
• Generally, the larger the noise
the stronger the response
• What is to be done?
– intuitively, most pixels in
images look quite a lot like
their neighbours
– this is true even at an edge;
along the edge they’re similar,
across the edge they’re not
– suggests that smoothing the
image should help, by forcing
pixels different to their
neighbours (=noise pixels?) to
look more like neighbours
Computer Vision : CISC 4/689
Finite differences responding to noise
Increasing noise ->
(this is zero mean additive gaussian noise)
Computer Vision : CISC 4/689
The response of a linear filter to noise
•
•
Do only stationary independent
additive Gaussian noise with zero
mean (non-zero mean is easily
dealt with)
Mean:
– output is a weighted sum of inputs
– so we want mean of a weighted
sum of zero mean normal random
variables
– must be zero
•
Variance:
– recall
• variance of a sum of random
variables is sum of their
variances
• variance of constant times
random variable is constant^2
times variance
– then if  is noise variance and
kernel is K, variance of response is
Remember: Gaussian is symmetric,
product of 2 Gaussians is Gaussian
2
 2  K u,v
u,v
Computer Vision : CISC 4/689
Filter responses (of noise) are correlated
• over scales similar to the scale of the filter
• Filtered noise is sometimes useful
– looks like some natural textures, can be used to simulate fire, etc.
Computer Vision : CISC 4/689
Computer Vision : CISC 4/689
Computer Vision : CISC 4/689
Computer Vision : CISC 4/689
Smoothing reduces noise
• Generally expect pixels to “be
like” their neighbours
– surfaces turn slowly
– relatively few reflectance
changes
• Generally expect noise
processes to be independent
from pixel to pixel
• Implies that smoothing
suppresses noise, for
appropriate noise models
• Scale
– the parameter in the symmetric
Gaussian
– as this parameter goes up,
more pixels are involved in the
average
– and the image gets more
blurred
– and noise is more effectively
suppressed
Computer Vision : CISC 4/689
The effects of smoothing
Each row shows smoothing
with gaussians of different
width; each column shows
different realisations of
an image of gaussian noise.
Computer Vision : CISC 4/689
Gradients and edges
• Points of sharp change in an
image are interesting:
–
–
–
–
change in reflectance
change in object
change in illumination
noise
• Sometimes called edge points
• General strategy
– determine image gradient
– now mark points where
gradient magnitude is
particularly large wrt
neighbours (ideally, curves of
such points).
Computer Vision : CISC 4/689
The Gradient and Edges
• Consider image intensities as a 2-D height function I(x,
Then the image gradient is the vector field defined by:
• Definition of an edge
– Line segment separating regions of contrasting intensity
– Location: Where gradient magnitude is high
– Direction: Orthogonal to the gradient
Computer Vision : CISC 4/689
y).
Edge Causes
• Depth discontinuity
• Surface orientation
discontinuity
• Reflectance discontinuity (i.e.,
change in surface material
properties)
• Illumination discontinuity (e.g.,
shadow)
Computer Vision : CISC 4/689
Edge Detection
•
An edge point can be regarded as a
point in an image where a discontinuity
(in gradient) occurs across some line. A
discontinuity may be classified as one
of five types
•
Searching for Edges:
Gradient Discontinuity
-- where the gradient of the pixel values changes
across a line.
This type of discontinuity can be classed as roof edges,
ramp edges convex edges concave edges, by noting the
sign of the component of the gradient perpendicular to
the edge on either side of the edge. Ramp edges have the
same signs in the gradient components on either side of
the discontinuity, while roof edges have opposite signs
in the gradient components.
A Jump or Step Discontinuity
-- where pixel values themselves change suddenly
across some line.
A Bar Discontinuity
-- where pixel values rapidly increase then decrease
again (or vice versa) across some line.
– Filter: Smooth image
– Enhance: Apply numerical
derivative approximation
Source: http://homepages.inf.ed.ac.uk/rbf/CVonline/
– Detect: Threshold to find strong
LOCAL_COPIES/MARSHALL/node28.html
edges
– Localize/analyze: Reject spurious
edges, include weak but justified
edges
Computer Vision : CISC 4/689
Step edge detection:
First Derivative Operators
• Method: Differentiate and find extrema
• Examples
– Sobel operator (Matlab: edge(I, ‘sobel’))
– Prewitt, Roberts cross
– Derivative of Gaussian
1
0
-1
1
2
1
2
0
-2
0
0
0
1
0
-1
-1 -2 -1
Sobel x
Sobel y
Computer Vision : CISC 4/689
Book uses this
format
Sobel Edge Filtering Example
1
0
-1
2
0
-2
1
0
-1
Rotate
-1
0
1
-2
0
2
-1
0
1
0
0
2
2
0
0
2
2
0
0
2
2
0
0
2
2
Computer Vision : CISC 4/689
-1
0
1
-2
0
2
-1
0
1
Step 1
-1
0
1
-2
02
02
2
2
-1
02
01
2
2
0
0
1
2
0
0
2
2
0
Computer Vision : CISC 4/689
0
0
2
2
0
0
2
2
0
0
2
2
0
0
2
2
-1
0
1
-2
0
2
-1
0
1
Step 2
-1
0
1
02
02
42
2
02
01
23
2
0
0
2
2
0
0
2
2
0
Computer Vision : CISC 4/689
6
0
0
2
2
0
0
2
2
0
0
2
2
0
0
2
2
-1
0
1
-2
0
2
-1
0
1
Step 3
-1
0
1
0
02
02
43
0
01
03
23
0
0
2
2
0
0
2
2
0
Computer Vision : CISC 4/689
6
0
0
2
2
0
0
2
2
0
0
2
2
0
0
2
2
6
-1
0
1
-2
0
2
-1
0
1
Step 4
-1
0
1
0
0
2
-4
30
2
0
0
3
-2
30
1
0
0
2
2
0
0
2
2
0
Computer Vision : CISC 4/689
6
0
0
2
2
0
0
2
2
0
0
2
2
0
0
2
2
6
-6
edge
effect
from
zeropadding
Sobel Edge Filtering Example: Result
0
6
6
-6
0
8
8
-8
0
8
8
-8
0
6
6
-6
(pad with zeroes again, the boundary)
and then we threshold…
Computer Vision : CISC 4/689
Sobel Edge Detection:
Gradient Approximation
Note anisotropy of edge finding
1
0
-1
2
0
-2
1
0
-1
Horizontal
1
2
1
0
0
0
Computer Vision : CISC 4/689
-1 -2 -1
Vertical
Sobel
• These can then be combined together to find the absolute magnitude of
the gradient at each point and the orientation of that gradient. The
gradient magnitude is given by:
• an approximate magnitude is computed using:
which is much faster to compute.
• The angle of orientation of the edge (relative to the pixel grid) giving
rise to the spatial gradient is given by:
In this case, orientation 0 is taken to mean that the direction of maximum
contrast from black to white runs from left to right on the image, and
other angles are measured anti-clockwise from this.
Computer Vision : CISC 4/689
Derivative of Gaussian
Computer Vision : CISC 4/689