Transcript Document

Motivating Applications
HDTV
Artistic reproduction
Internet video
Widescreen movie
EE565 Advanced Image Processing
Copyright Xin Li 2008
1
Image Interpolation

Image Interpolation




Importance of geometry
Edge directed interpolation
PDE-based techniques
Applications



Super-resolution
Inpainting (error concealment)
Compressive sensing
EE565 Advanced Image Processing
Copyright Xin Li 2008
2
Importance of Geometry



It is geometry that distinguishes
image signals (2D) from audio and
speech (1D)
2D image is the projection of 3D
geometry (a seemingly-trivial
statement)
Geometric elements and properties:
edge (location and orientation),
object (convexity) … …
EE565 Advanced Image Processing
Copyright Xin Li 2008
3
Geometric Constraint of Edges
single-edge image
3D visualization
Observations:
• Across the edge orientation, intensity field evolves fast
• Along the edge orientation, intensity field is homogeneous
EE565 Advanced Image Processing
Copyright Xin Li 2008
4
Why geometry is difficult?


Geometry is embedded in the array
of image pixels, which is not
straightforward to be exploited by
any linear operations (filtering or
transform)
Image signals form a manifold in an
extremely high-dimensional space
(What is a manifold? Think of a
string in the real world, it is a 3D
object and a 1D manifold)
EE565 Advanced Image Processing
Copyright Xin Li 2008
5
Interpolation Problem
down
sampling
original
high-resolution
image X
low-resolution
image Y
interpolated
high-resolution
image Z
How to make Z as close to X as possible?
EE565 Advanced Image Processing
Copyright Xin Li 2008
6
bilinear Interpolation Review
(e+w)/2
(n+s)/2
(ne+nw+se+sw)/4
EE565 Advanced Image Processing
Copyright Xin Li 2008
7
Why bilinear is bad?


Edge blurring
Jagged artifacts
Jagged artifacts
Edge blurring
Z
X
X
EE565 Advanced Image Processing
Copyright Xin Li 2008
Z
8
Where does image quality degradation
come from?


Violation of geometric constraint
Keep in mind an edge could be
arbitrarily oriented
Across the edge orientation,
intensity field evolves slower
Edge blurring
Along the edge orientation,
intensity field is inhomogeneous
Jagged artifacts
EE565 Advanced Image Processing
Copyright Xin Li 2008
9
Ideas to Do Better

Recognize the importance of edges



Location: where are they?
Orientation: how are they oriented?
Identify appropriate mathematical
tools for exploiting geometric
constraints


Heuristics
Statistical vs. PDE
EE565 Advanced Image Processing
Copyright Xin Li 2008
10
Heuristics: Edge Location


Intuitively, edges are characterized
by fast intensity value variation (i.e.,
large gradient)
To avoid edge blurring, we need to
spatially adaptively adjust
interpolation coefficients
Question: how to incorporate gradient cue into a spatially
adaptive interpolation scheme?
EE565 Advanced Image Processing
Copyright Xin Li 2008
11
Desired Improvement
X
Y
Zbil
Zesi
horizontal
edge
vertical
edge
EE565 Advanced Image Processing
Copyright Xin Li 2008
12
Here Comes the Trick
Step 1: interpolate the missing pixels along the diagonal
a
b
x
c
black or white?
d
Since |a-c|=|b-d|
x has equal probability
of being black or white
Step 2: interpolate the other half missing pixels
a
x
d
b
Since |a-c|>|b-d|
x=(b+d)/2=black
c
EE565 Advanced Image Processing
Copyright Xin Li 2008
13
Edge Sensitive Interpolation (ESI)
a
b
x
c
If |a-c|>|b-d|
x=(b+d)/2
a
x
d
d
b
c
If |a-c|=|b-d|
If |a-c|<|b-d|
x=min or max
x=(a+c)/2
In simple words, we want to avoid interpolate along the
direction which has a large gradient
EE565 Advanced Image Processing
Copyright Xin Li 2008
14
Experiment Result
Bilinear interpolation
Edge sensitive interpolation
EE565 Advanced Image Processing
Copyright Xin Li 2008
15
Heuristics: Edge Orientation

Can we do better?




Yes!
Gradient is only a first-order
characteristics of edge location
ESI makes binary decision with two
orthogonal directions
How to do better?

We need some mathematical tool that
can work with arbitrary edge
orientation
EE565 Advanced Image Processing
Copyright Xin Li 2008
16
Motivation
y
pattern
.
:
(-2,4)
(-1,2)
(0,0)
x
(1,-2)
:
.
Along the edge orientation,
We observe repeated pattern
EE565 Advanced Image Processing
Copyright Xin Li 2008
17
Geometric Duality
down
sampling
same pattern
EE565 Advanced Image Processing
Copyright Xin Li 2008
18
Bridge across the resolution
2j-2
2j
2j+2
High-resolution
2i-2
2i
2i+2
Low-resolution
Cov(X2i,2j,X2i+k,2j+l)≈Cov(X2i,2j,X2i+2k,2j+2l)
(k,l)={(0,1),(1,1),(1,0),(1,-1),(0,-1),(-1,-1),(-1,0),(-1,1)}
EE565 Advanced Image Processing
Copyright Xin Li 2008
19
Least-Square (LS) Method

Solve square linear system
1
nn nn
Ynn  Ann X nn  X nn  A Y

Solve over-determined system
Ym1  Amn X n1  X  ( AT A)1 ( AT Y )
EE565 Advanced Image Processing
Copyright Xin Li 2008
20
LS-based estimation
8
X   ai X i
X 3 X2
X4
X5
X
i 1
X1
X8
X6 X7
For all pixels in 7x7 window,
we can write an equation like
above, which renders an
over-determined system
with 49 equations and 8
unknown variables
Use LS method to solve
EE565 Advanced Image Processing
Copyright Xin Li 2008
21
Step 1: Interpolate diagonal pixels
1
4
0
1
Implementation:
4
-Use {a1,a2,a3,a4} to
interpolate the pixel
0 at the high resolution
0
2
2
-Formulate LS estimation
problem with pixels at
low resolution and solve
{a1,a2,a3,a4}
3
3
EE565 Advanced Image Processing
Copyright Xin Li 2008
22
Step 2: Interpolate the Other Half
1
0
2
Implementation:
0
2
3
4
1
4
-Formulate LS estimation
problem with pixels at
low resolution and solve
{a1,a2,a3,a4}
-Use {a1,a2,a3,a4} to
interpolate the pixel
0 at the high resolution
3
EE565 Advanced Image Processing
Copyright Xin Li 2008
23
Experiment Result
bilinear
Edge directed interpolation
EE565 Advanced Image Processing
Copyright Xin Li 2008
24
After Thoughts

Pro


Con


Improve visual quality dramatically
Computationally expensive
Further optimization

Translation invariant derivation of
interpolation coefficients a’s
EE565 Advanced Image Processing
Copyright Xin Li 2008
25