Transcript EE 7730: Lecture 1 - Computer Engineering

EE 7730: Image Analysis I
Introduction
EE 7730






Dr. Bahadir K. Gunturk
Office: EE 225
Email: [email protected]
Tel: 8-5621
Office Hours: MW 2:40 – 4:30
Class Hours: MWF 1:40 – 2:30 (CEBA-3129)
Bahadir K. Gunturk
2
EE 7730




We will learn the fundamentals of digital image processing,
computer vision, and digital video processing
Lecture slides, problems sets, solutions, study materials, etc. will be
posted on the class website. [www.ece.lsu.edu/gunturk/EE7730]
Textbook is not required.
References:





Gonzalez/Woods, Digital Image Processing, Prentice-Hall, 2/e.
Forsyth/Ponce, Computer Vision: A Modern Approach, Prentice-Hall.
Duda, Hart, and Stork, Pattern Classification, John Wiley&Sons, 2001.
Tekalp, Digital Video Processing, 1995
Jain, Fundamentals of Digital Image Processing, Prentice-Hall.
Bahadir K. Gunturk
3
Grading Policy

Your grade will be based on




Problem Sets




Problem Sets + Semester Project: 35%
Midterm: 30%
Final: 35%
Theoretical problems and MATLAB assignments
4-5 Problem Sets
Individually or in two-person teams
Semester Project

Each student will give a 15 minute presentation
Bahadir K. Gunturk
4
EE 7740 Image Analysis II

Semester Project




Possible project topics will be provided in a month
Projects will be done individually
Projects will involve MATLAB or C/C++ implementation
Each student will give a 15 minute presentation at the end of
the semester
Bahadir K. Gunturk
5
EE 7740 Image Analysis II

Image Analysis I - Outline









Digital image fundamentals
2D Fourier transform, sampling, Discrete Cosine Transfrom
Image enhancement
Human visual system and color image processing
Image restoration
Image compression
Image segmentation
Morphology
Introduction to digital video processing
Bahadir K. Gunturk
6
Digital Image Acquisition
Sensor array



Bahadir K. Gunturk
When photons strike, electron-hole pairs are generated
on sensor sites.
Electrons generated are collected over a certain period
of time.
The number of electrons are converted to pixel values.
(Pixel is short for picture element.)
7
Digital Image Acquisition
Two types of quantization:
1. There are finite number of
pixels. (Spatial resolution)
2. The amplitude of pixel is
represented by a finite
number of bits. (Gray-scale
resolution)
Bahadir K. Gunturk
8
Digital Image Acquisition
Bahadir K. Gunturk
9
Digital Image Acquisition
•
•
•
•
•
•
Bahadir K. Gunturk
256x256 - Found on very cheap
cameras, this resolution is so low that
the picture quality is almost always
unacceptable. This is 65,000 total pixels.
640x480 - This is the low end on most
"real" cameras. This resolution is ideal
for e-mailing pictures or posting pictures
on a Web site.
1216x912 - This is a "megapixel" image
size -- 1,109,000 total pixels -- good for
printing pictures.
1600x1200 - With almost 2 million total
pixels, this is "high resolution." You can
print a 4x5 inch print taken at this
resolution with the same quality that you
would get from a photo lab.
2240x1680 - Found on 4 megapixel
cameras -- the current standard -- this
allows even larger printed photos, with
good quality for prints up to 16x20
inches.
4064x2704 - A top-of-the-line digital
camera with 11.1 megapixels takes
pictures at this resolution. At this setting,
you can create 13.5x9 inch prints with
no loss of picture quality.
10
Matrix Representation of Images

A digital image can be written as a matrix
x[0,1]
 x[0, 0]
 x[1, 0]
x[1,1]
x[n1 , n2 ]  


 x[ M  1, 0]
x[0, N  1] 
x[1, N  1] 


x[ M  1, N  1] MxN
35 45 20 
 43 64 52 


10 29 39 
Bahadir K. Gunturk
11
Image Resolution
Bahadir K. Gunturk
12
Bit Depth – Grayscale Resolution
8 bits
7 bits
6 bits
5 bits
Bahadir K. Gunturk
13
Bit Depth – Grayscale Resolution
4 bits
2 bits
Bahadir K. Gunturk
3 bits
1 bit
14
Digital Color Images

xR [n1 , n2 ]
xG [n1, n2 ]
xB [n1 , n2 ]
Bahadir K. Gunturk
15
Video

x[n1 , n2 , n3 ]
n1 = vertical position
n2 = horizontal position
n3 = frame number
~24 frames per second.
Bahadir K. Gunturk
16
Why do we process images?





To facilitate their storage and transmission
To prepare them for display or printing
To enhance or restore them
To extract information from them
To hide information in them
Bahadir K. Gunturk
17
Image Processing Example

Image Restoration
Original image
Bahadir K. Gunturk
Blurred
Restored by Wiener
filter
18
Image Processing Example

Noise Removal
Noisy image
Bahadir K. Gunturk
Denoised by Median filter
19
Image Processing Example

Image Enhancement
Histogram
equalization
Bahadir K. Gunturk
20
Image Processing Example

Artifact Reduction in Digital Cameras
Original scene
Bahadir K. Gunturk
Captured by a digital camera
Processed to reduce artifacts
21
Image Processing Example

Image Compression
Original image
64 KB
Bahadir K. Gunturk
JPEG compressed
15 KB
JPEG compressed
9 KB
22
Image Processing Example

Object Segmentation
“Rice” image
Bahadir K. Gunturk
Edges detected using Canny
filter
23
Image Processing Example

Resolution Enhancement
Bahadir K. Gunturk
24
Image Processing Example

Watermarking
Original image
Watermarked
image
Generate
watermark
Hidden message
Secret key
Bahadir K. Gunturk
25
Image Processing Example

Face Recognition
Search in the
database
Surveillance video
Bahadir K. Gunturk
26
Image Processing Example

Fingerprint Matching
Bahadir K. Gunturk
27
Image Processing Example

Segmentation
Bahadir K. Gunturk
28
Image Processing Example

Texture Analysis and Synthesis
Photo
Pattern repeated
Bahadir K. Gunturk
Computer generated
29
Image Processing Example

Face detection and tracking
http://www-2.cs.cmu.edu/~har/faces.html
Bahadir K. Gunturk
30
Image Processing Example

Face Tracking
Bahadir K. Gunturk
31
Image Processing Example

Object Tracking
Bahadir K. Gunturk
32
Image Processing Example

Virtual Controls
Bahadir K. Gunturk
33
Image Processing Example

Visually Guided Surgery
Bahadir K. Gunturk
34
Cameras


First camera was invented in 16th century.
It used a pinhole to focus light rays onto a wall or
translucent plate.


Bahadir K. Gunturk
Take a box, prick a small hole in one
of its sides with a pin, and then
replace the opposite side with a
translucent plate.
Place a candle on the pinhole side,
you will see an inverted image of the
candle on the translucent plate.
35
Perspective Projection

Perspective projection equations
x' y' z'
 
x
y
z
Bahadir K. Gunturk
36
Pinhole Camera Model


If the pinhole were really reduced to a point, exactly one light ray would
pass through each point in the image plane.
In reality, each point in the image place collects light from a cone of rays.
Bahadir K. Gunturk
37
Pinhole Cameras
Pinhole too big many directions are
averaged, blurring the
image
Pinhole too small diffraction effects blur
the image
Bahadir K. Gunturk
38
Cameras With Lenses


Most cameras are equipped with lenses.
There are two main reasons for this:


To gather light. For an ideal pinhole, a single light ray would
reach each point the image plane. Real pinholes have a finite
size, so each point in the image plane is illuminated by a cone
of light rays. The larger the hole, the wider the cone and the
brighter the image => blurry pictures. Shrinking the pinhole
produces sharper images, but reduces the amount of light
and may introduce diffraction effects.
To keep the picture in sharp focus while gathering light from a
large area.
Bahadir K. Gunturk
39
Compound Lens Systems
Bahadir K. Gunturk
40
Real Lenses

Rays may not focus at a single point.
Spherical aberration
Spherical aberration can be eliminated completely by designing aspherical lenses.
Bahadir K. Gunturk
41
Real Lenses


The index of refraction is a function of wavelength.
Light at different wavelengths follow different paths.
Chromatic aberration
Bahadir K. Gunturk
42
Real Lenses
Chromatic Aberration
Bahadir K. Gunturk
43
Real Lenses

Special lens systems using two or more pieces of glass with different
refractive indeces can reduce or eliminate this problem. However, not
even these lens systems are completely perfect and still can lead to
visible chromatic aberrations.
Bahadir K. Gunturk
44
Real Lenses
Causes of distortion

Barrel Distortion & Pincushion Distortion
Stop (Aperture)
Chief ray
(normal)
Bahadir K. Gunturk
45
Real Lenses

Barrel Distortion & Pincushion Distortion
Distorted
Corrected
http://www.vanwalree.com/optics/distortion.html
http://www.dpreview.com/learn/?/Image_Techniques/Barrel_Distortion_Correction_01.htm
Bahadir K. Gunturk
46
Real Lenses
Vignetting effect in a two-lens system. The shaded part of
the beam never reaches the second lens. The brightness
drop in the image perimeter.
Bahadir K. Gunturk
47
Real Lenses
Optical vignetting example. Left: f/1.4. Right: f/5.6.
f-number
focal length to diameter ratio
Bahadir K. Gunturk
48
Real Lenses
Long exposure time
Short exposure time
Bahadir K. Gunturk
49
Real Lenses
Flare
Hood may prevent flares
Bahadir K. Gunturk
50
Real Lenses
Flare
Bahadir K. Gunturk
51
Compound Lens Systems
http://www.dpreview.com/learn/?/glossary/
http://www.cartage.org.lb/en/themes/Sciences/Physics/Optics/Optical/Lens/Lens.htm
http://www.vanwalree.com/optics.html
Bahadir K. Gunturk
52
Digital Camera Pipeline
(from Katz and Gentile)
Auto-exposure algorithms measure brightness over discrete scene regions to
compensate for overexposed or underexposed areas by manipulating shutter
speed and/or aperture size. The net goals here are to maintain relative
contrast between different regions in the image and to achieve a good overall
quality.
Bahadir K. Gunturk
53
Digital Camera Pipeline
Auto-focus algorithms divide into two categories. Active methods use infrared
or ultrasonic emitters/receivers to estimate the distance between the camera
and the object being photographed. Passive methods, on the other hand,
make focusing decisions based on the received image in the camera.
Bahadir K. Gunturk
54
Digital Camera Pipeline
Lens distortion correction
This set of algorithms accounts for the physical properties of lenses that warp
the output image compared to the actual scene the user is viewing. Different
lenses can cause different distortions; for instance, wide-angle lenses create
a "barrel distortion", while telephoto lenses create a "pincushion distortion“.
Bahadir K. Gunturk
55
Digital Camera Pipeline
Vignetting (shading distortion) reduces image brightness in the area around
the lens. Chromatic aberration causes color fringes around an image. The
media processor needs to mathematically transform the image in order to
correct for these distortions.
Bahadir K. Gunturk
56
Digital Camera Pipeline
Sensor's output needs to be gamma-corrected to account for eventual display,
as well as to compensate for nonlinearities in the sensor's capture response.
Bahadir K. Gunturk
57
Digital Camera Pipeline
Image stability compensation, or hand-shaking correction is another area of
preprocessing. Here, the processor adjusts for the translational motion of the
received image, often with the help of external transducers that relate the
real-time motion profile of the sensor.
Bahadir K. Gunturk
58
Digital Camera Pipeline
White balance is another important stage of preprocessing. When we look at
a scene, regardless of lighting conditions, our eyes tend to normalize
everything to the same set of natural colors. For instance, an apple looks
deep red to us whether we're indoors under fluorescent lighting, or outside in
sunny weather. However, an image sensor's "perception" of color depends
largely on lighting conditions, so it needs to map its acquired image to appear
natural in its final output. This mapping can be done either manually or
automatically.
Bahadir K. Gunturk
59
Digital Camera Pipeline
Demosaicking (Bayer interpolation) estimates missing color samples in singlechip cameras.
Bahadir K. Gunturk
60
Digital Camera Pipeline
In this stage, the interpolated RGB image is transformed to the targeted
output color space (if not already in the right space). For compression or
display to a television, this will usually involve an RGBYCbCr matrix
transformation, often with another gamma correction stage to accommodate
the target display. The YCbCr outputs may also be chroma subsampled at this
stage to the standard 4:2:2 format for color bandwidth reduction with little
visual impact.
Bahadir K. Gunturk
61
Digital Camera Pipeline
Postprocessing
In this phase, the image is perfected via a variety of filtering operations before
being sent to the display and/or storage media. For instance, edge
enhancement, pixel thresholding for noise reduction, and color-artifact
removal are all common at this stage.
Bahadir K. Gunturk
62
Digital Camera Pipeline
Display / Compress / Store
Once the image itself is ready for viewing, the image pipe branches off in two
different directions. In the first, the postprocessed image is output to the target
display, usually an integrated LCD screen (but sometimes an NTSC/PAL
television monitor, in certain camera modes). In the second, the image is sent
to the media processor's compression algorithm, where industry-standard
compression techniques (JPEG, for instance) are applied before the picture is
stored locally in some storage medium (e.g., Flash memory card).
Bahadir K. Gunturk
63
Review: Linear Systems

We define a system as a unit that converts an input function
into an output function.
Independent System operator
variable
Bahadir K. Gunturk
64
Linear Systems

Let
where fi(x) is an arbitrary input in the class of all inputs
{f(x)}, and gi(x) is the corresponding output.

If
Then the system H is called a linear system.

A linear system has the properties of additivity and
homogeneity.
Bahadir K. Gunturk
65
Linear Systems

The system H is called shift invariant if
for all fi(x) {f(x)} and for all x0.

This means that offsetting the independent variable of the
input by x0 causes the same offset in the independent
variable of the output. Hence, the input-output relationship
remains the same.
Bahadir K. Gunturk
66
Linear Systems

The operator H is said to be causal, and hence the system
described by H is a causal system, if there is no output
before there is an input. In other words,

A linear system H is said to be stable if its response to any
bounded input is bounded. That is, if
where K and c are constants.
Bahadir K. Gunturk
67
Linear Systems

A unit impulse function, denoted (a), is defined by the
expression
(a)
(x-a)
a
x
Bahadir K. Gunturk
68
Linear Systems

A unit impulse function, denoted (a), is defined by the
expression
Then
Bahadir K. Gunturk
69
Linear Systems

The term
is called the impulse response of H.

From the previous slide

It states that, if the response of H to a unit impulse [i.e., h(x,
)], is known, then response to any input f can be computed
using the preceding integral. In other words, the response of a
linear system is characterized completely by its impulse
response.
Bahadir K. Gunturk
70
Linear Systems

If H is a shift-invariant system, then
and the integral becomes

This expression is called the convolution integral. It states that
the response of a linear, fixed-parameter system is completely
characterized by the convolution of the input with the system
impulse response.
Bahadir K. Gunturk
71
Linear Systems

Convolution of two functions is defined as

f ( x)* h( x) 

f ( )h( x   )d


In the discrete case
f [n]* h[n] 


f [m]h[n  m]
m 
Bahadir K. Gunturk
72
Linear Systems

In the 2D discrete case
f [n1 , n2 ]**h[n1 , n2 ] 
h[n1 , n2 ]
Bahadir K. Gunturk


 
m1  m2 
f [m1 , m2 ]h[n1  m1 , n2  m2 ]
is a linear filter.
73
Example
*
Bahadir K. Gunturk
1 1 1
1

1
1
1

9
1 1 1
=
74
Example
*
Bahadir K. Gunturk
1 1 1
1 8 1


1 1 1
=
75
Try MATLAB
f=imread(‘saturn.tif’);
figure; imshow(f);
[height,width]=size(f);
f2=f(1:height/2,1:width/2);
figure; imshow(f2);
[height2,width2=size(f2);
f3=double(f2)+30*rand(height2,width2);
figure;imshow(uint8(f3));
h=[1 1 1 1; 1 1 1 1; 1 1 1 1; 1 1 1 1]/16;
g=conv2(f3,h);
figure;imshow(uint8(g));
Bahadir K. Gunturk
76