Transcript EE 7730: Lecture 1 - Louisiana State University

EE 7700
Demosaicking Problem in Digital Cameras
Multi-Chip Digital Camera


To produce a color image, at least three spectral
components are needed at each pixel.
One approach is to use beam-splitters and multiple chips.
Lens
Scene
Beamsplitters
Bahadir K. Gunturk
Spectral
filters
Sensors
2
Single-Chip Digital Camera


Multi-chip approach is expensive. Precise chip alignment is
required.
The alternative is to use a color filter array.
Lens
Color filter
array
Sensors
Scene
Bahadir K. Gunturk
3
Single-Chip Digital Camera


The missing color samples must be estimated to produce
the full color image.
Since a mosaic of samples are available, this estimation
(interpolation) process is called demosaicking.
Bahadir K. Gunturk
4
Single-Chip Digital Camera

Images suffer from color artifacts when the samples are not
estimated correctly.
Original image
Bahadir K. Gunturk
Bilinearly interpolated
from CFA-filtered samples
5
Demosaicking Approaches


Non-Adaptive Single-Channel Interpolation: Interpolate
each color channel separately using a standard technique,
such as nearest-neighbor interpolation, bilinear
interpolation, etc.
Edge-Directed Interpolation: Estimate potential edges,
avoid interpolating across the edges.
Edge-directed interpolation
3
1
x
4
Bahadir K. Gunturk
2
1.Calculate horizontal gradient ΔH = |G1 – G2|
2.Calculate vertical gradient ΔV = |G3 – G4|
3.If ΔH > ΔV,
Gx = (G3 + G4)/2
Else if ΔH < ΔV,
Gx = (G1 + G2)/2
Else
Gx = (G1 + G2 + G3 + G4)/4
6
Demosaicking Approaches

Edge-Directed Interpolation: Based on the assumption that
color channels have similar texture, various edge detectors
can be used.
Edge-directed interpolation
1
2
3
4
5
8
9
Bahadir K. Gunturk
6
7
1.
2.
3.
Calculate horizontal gradient ΔH = | (R3 + R7)/2 – R5 |
Calculate vertical gradient ΔV = | (R1 + R9)/2 – R5 |
If ΔH > ΔV,
G5 = (G2 + G8)/2
Else if ΔH < ΔV,
G5 = (G4 + G6)/2
Else
G5 = (G2 + G8 + G4 + G6)/4
7
Demosaicking Approaches

Constant-Hue-Based Interpolation: Hue does not change
abruptly within a small neighborhood.



Interpolate green channel first.
Interpolate hue (defined as either color differences or color
ratios).
Estimate the missing (red/blue) from the interpolated hue.
Interpolate
Red
Green
Bahadir K. Gunturk
Interpolate
d Red
Interpolate
8
Demosaicking Approaches

Edge-Directed Interpolation of Hue: It is a combination of
edge-directed interpolation and constant-hue-based
interpolation. Hue is interpolated as in constant-hue-based
interpolation approach, but this time, hue is interpolated
based on the edge directions (as in the edge-directed
interpolation algorithm).
Bahadir K. Gunturk
9
Demosaicking Approaches

Using Laplacian For Enhancement: Use the second-order
gradients of red/blue channels to enhance green channel.
1.
2.
3.
1
2
3
4
5
8
9
Bahadir K. Gunturk
6
7
Calculate horizontal gradient ΔH = |G4 – G6| + |R5 – R3 + R5 – R7|
Calculate vertical gradient ΔV = |G2 – G8| + |R5 – R1 + R5 – R9|
If ΔH > ΔV,
G5 = (G2 + G8)/2 + (R5 – R1 + R5 – R9)/4
Else if ΔH < ΔV,
G5 = (G4 + G6)/2 + (R5 – R3 + R5 – R7)/4
Else
G5 = (G2 + G8 + G4 + G6)/4 + (R5 – R1 + R5 – R9 + R5 – R3 + R5 – R7)/8
10
Aliasing
f2
Frequency spectrum of an image:
fm
f1
After CFA sampling:
f2
f2
f1
Green channel
Bahadir K. Gunturk
f1
Red/Blue channel
11
Demosaicking Approach

Alias Cancelling: Based on the assumption that red, green,
and blue channels have similar frequency components, the
high-frequency components of red and blue channels are
replaced by the high-frequency components of green
channel.
f2
f1
Red/Blue channel
Bahadir K. Gunturk
12
Experiment
HL
Full
Red/Green/Blue
channels
Subband
decomposition
LL
LL
HL
HL
LL
HH
LH
LH
LH
CFA
Sampling
Interpolate
HL
Subband
decomposition
LL
HL
LL
HH
LH
Bahadir K. Gunturk
LL
HL
LH
LH
13
Constraint Sets
 Detail Constraint Set: Detail subbands of the red and blue
channels must be similar to the detail subbands of the green
channel.
GHL
LH
HL
HL
HH
HH
RHL
LH

 R(n1 , n2 ) : Rk (n1 , n2 )  Gk (n1 , n2 )  T (n1 , n2 ), 
Cd  



 for k  HL, LH , HH

Bahadir K. Gunturk
14
Constraint Sets
 Observation Constraint Set: Interpolated channels must be
consistent with the observed data.
Sensors
CFA
O(n1 , n2 )
R
Co  R(n1, n2 ) : R(n1, n2 )  O(n1, n2 ),  (n1, n2 ) R 
Bahadir K. Gunturk
15
Projection Operations
 Projection onto the Detail Constraint Set:
 Decompose the color channels.
 Update the detail subbands of red and blue channels.
GHL (n1 , n2 )
HL
HH
LH
RHL (n1 , n2 )
GHL (n1 , n2 )  T (n1, n2 )
 Apply synthesis filters to reconstruct back the channels.
Bahadir K. Gunturk
16
Projection Operations
 Projection onto the Observation Constraint Set:
 Insert the observed data to their corresponding positions.
Sensors
CFA
O(n1 , n2 )
Bahadir K. Gunturk
17
Alternating Projections Algorithm
Samples of
color channels
Initial
interpolation
Projection onto the
detail constraint set
g0
h0
h1
Update
Projection onto the
observation constraint set
Insert the
observed data
g1
Iteration
Bahadir K. Gunturk
18
Results
Original
Hibbard 1995
Laroche and Prescott 1994
Hamilton and Adams 1997
Kimmel 1999
Gunturk 2002
Bahadir K. Gunturk
19
Results
Original
Hibbard
1995
Laroche
and
Prescott
1994
Hamilton
and
Adams
1997
Kimmel
1999
Gunturk
2002
Bahadir K. Gunturk
20
Previous Methods
[Gunturk02]
Gunturk et al, “Demosaicking: Color Filter Array Interpolation in Single-Chip Digital
Cameras,” to appear in IEEE Signal Processing Magazine.
Bahadir K. Gunturk
21
References
 [Gunturk02] Gunturk et al, “Color Plane Interpolation Using Alternating
Projections,” IEEE Trans. Image Processing, 2002.
[Hibbard 1995] R. H. Hibbard, “Apparatus and method for
adaptively interpolating a full color image utilizing luminance
gradients,” U.S. Patent 5,382,976, January, 1995.
 [Laroche and Prescott 1994] C. A. Laroche and M. A. Prescott,
“Apparatus and method for adaptively interpolating a full color
image utilizing chrominance gradients,” U.S. Patent 5,373,322,
December, 1994.
 [Hamilton and Adams 1997] J. F. Hamilton Jr. and J. E. Adams,
“Adaptive color plane interpolation in single sensor color electronic
camera,” U.S. Patent 5,629,734, May, 1997.
 [Kimmel 1999] R. Kimmel, “Demosaicing: Image reconstruction
from CCD samples,” IEEE Trans. Image Processing, vol. 8, pp.
1221-1228, 1999.
Bahadir K. Gunturk
22