Transcript Document 7345632

Super-Resolution
Digital Photography
CSE558, Spring 2003
Richard Szeliski
Super-resolution
• convolutions, blur, and de-blurring
• Bayesian methods
• Wiener filtering and Markov Random Fields
• sampling, aliasing, and interpolation
• multiple (shifted) images
• prior-based methods
• MRFs
• learned models
• domain-specific models (faces)- Gary
3/7/2003
Super-Resolution
2
Linear systems
Basic properties
• homogeneity T[a X]
= a T[X]
• additivity
T[X1+X2] = T[X1]+T[X2]
• superposition T[aX1+bX2] = aT[X1]+bT[X2]
Linear system  superposition
Examples:
• matrix operations (additions, multiplication)
• convolutions
3/7/2003
Super-Resolution
3
Signals and linear operators
Continuous
Discrete
Vector formI
I(x)
I[k] or Ik
Discrete linear operator
y=Ax
Continuous linear operator:
convolution integral
g(x) = sh(,x) f() d, h(,x): impulse response
g(x) = s h(-x) f() d= [f * h](x) shift invariant
3/7/2003
Super-Resolution
4
2-D signals and convolutions
Continuous
Discrete
I(x,y)
I[k,l] or Ik,l
2-D convolutions (discrete)
g[k,l] = m,n f[m,n] h[k-m,l-n]
= m,n f[m,n] h1[k-m]h2[l-n] separable
Gaussian kernel is separable and radial
h(x,y) = (22)-1exp-(x2+y2)/2
3/7/2003
Super-Resolution
5
Convolution and blurring
3/7/2003
Super-Resolution
6
Separable binomial low-pass filter
3/7/2003
Super-Resolution
7
Fourier transforms
Project onto a series of complex sinusoids
F[m,n] = k f[k,l] e-i 2(km+ln)
Properties:
• shifting
g(x-x0)  exp(-i 2fxx0)G(fx)
• differentiation dg(x)/dx  i 2fxG(fx)
• convolution
[f * g](x)  [F G] (fx)
3/7/2003
Super-Resolution
8
Blurring examples
Increasing amounts of blur + Fourier transform
3/7/2003
Super-Resolution
9
Sharpening
Unsharp mask (darkroom photography):
• remove some low-frequency content
y’ = y + s (y – g * y)
spatial (blur, sharp)
3/7/2003
Super-Resolution
freq (blur,sharp)
10
Sharpening - result
Unsharp mask: original, blur (σ=1),
sharp(s=0, 1, 2)
3/7/2003
Super-Resolution
11
Deconvolution
Filter by inverse of blur
• easiest to do in the Fourier domain
• problem: high-frequency noise amplification
3/7/2003
Super-Resolution
12
Bayesian modeling
Use prior model for image and noise
• y = g * x + n, x is original, y is blurred
• p(x|y) = p(y|x)p(x)
= exp(-|y – g*x|2/2σn-2) exp(-|x|2/2σx-2)
• -log p(x|y)  |y – g*x|2σn-2 + |x|2σx-2
where the norm || is summed squares over all
pixels
3/7/2003
Super-Resolution
13
Parseval’s Theorem
Energy equivalence in spatial ↔ frequency
domain
• |x|2 = |F(x)|2
• -log p(x|y)  |Y(f) – G(f)X(f)|2σn-2 + |X(f)|2σx-2
• least squares solution (∂/∂X = 0)
X(f) = G(f)Y(f) / [G2(f) + σn2/σx2]
3/7/2003
Super-Resolution
14
Wiener filtering
Optimal linear filter given noise and signal
statistics
• X(f) = G(f)Y(f) / [G2(f) + σn2/σx2]
• low frequencies:
X(f) ≈ G-1(f)Y(f)
boost by inverse gain (blur)
• high frequencies:
X(f) ≈ G(f) σn-2σx2 Y(f)
attenuate by blur (gain)
3/7/2003
Super-Resolution
15
Wiener filtering – white noise prior
Assume all frequencies equally likely
• p(x) ~ N(0,σx2)
• X(f) = G(f)Y(f) / [G2(f) + σn2/σx2]
• solution is too noisy in high frequencies
3/7/2003
Super-Resolution
16
Wiener filtering – pink noise prior
Assume frequency falloff (“natural statistics”)
• p(X(f)) ~ N(0,|f|-βσx2)
• X(f) = G(f)Y(f) / [G2(f) + |f|βσn2/σx2]
• greater attenuation at high frequencies
G(f)
3/7/2003
H(f)
Super-Resolution
17
Markov Random Field modeling
Use spatial neighborhood prior for image
i
• -log p(x) = ijCρ(xi-xj)
where ρ(v) is a robust norm:
•
•
•
•
j
ρ(v) = v2: quadratic norm ↔ pink noise
ρ(v) = |v|: total variation (popular with maths)
ρ(v) = |v|β: natural statistics
ρ(v) = v2,|v|: Huber norm
[Schultz, R.R.; Stevenson, IEEE TIP, 1996]
3/7/2003
Super-Resolution
18
MRF estimation
Set up discrete energy (quadratic or non-)
• -log p(x|y)  σn-2 |y – Gx|2 + ijCρ(xi-xj)
where G is sparse convolution matrix
• quadratic: solve sparse linear system
• non-quadratic: use sparse non-linear least
squares (Levenberg-Marquardt, gradient
descent, conjugate gradient, …)
3/7/2003
Super-Resolution
19
Sampling a signal
• sampling:
• creating a discrete signal from a continuous signal
• downsampling (decimation)
• subsampling a discrete signal
• upsampling
• introducing zeros between samples
• aliasing
• two sampled signals that differ in their original
form (many → one mapping)
3/7/2003
Super-Resolution
20
Sampling
interpolation
3/7/2003
Super-Resolution
21
Nyquist sampling theorem
Signal to be (down-) sampled must have a
bandwidth no larger than twice the sample
frequency
s = 2 / ns > 2 0
3/7/2003
Super-Resolution
22
Box filter (top hat)
3/7/2003
Super-Resolution
23
Ideal low-pass filter
3/7/2003
Super-Resolution
24
Simplified camera optics
1.
2.
3.
4.
Blur = pill-box*Bessel2 (diffr.) ≈ Gaussian
Integrate = box filter
Sample = produce single digital sample
Noise = additive white noise
3/7/2003
Super-Resolution
25
Aliasing
Aliasing (“jaggies” and “crawl”) is present if
blur amount < sampling (σ = 1)
• shift each image in previous pipeline by 1
3/7/2003
Super-Resolution
26
Aliasing - less
Less aliasing (“jaggies” and “crawl”) is present if
blur amount ~ sampling (σ = 2)
• shift each image in previous pipeline by 1
3/7/2003
Super-Resolution
27
Multi-image super-resolution
Exploit aliasing to recover frequencies above
Nyquist cutoff
• kσn-2 |yk – Gkx|2 + ijCρ(xi-xj)
where Gk are sparse convolution matrices
• quadratic: solve sparse linear system
• non-quadratic: use sparse non-linear least
squares (Levenberg-Marquardt, gradient
descent, conjugate gradient, …)
• projection onto convex sets (POCS)
3/7/2003
Super-Resolution
28
Multi-image super-resolution
Need:
• accurate (sub-pixel) motion estimates
(Wednesday’s lecture)
• accurate models of blur (pre-filtering)
• accurate photometry
• no (or known) non-linear pre-processing
(Bayer mosaics)
• sufficient images and low-noise relative to
amount of aliasing
3/7/2003
Super-Resolution
29
Prior-based Super-Resolution
“Classical” non-Gaussian priors:
• robust or natural statistics
• maximum entropy (least blurry)
• constant colors (black & white images)
3/7/2003
Super-Resolution
30
Example-based Super-Resolution
William T. Freeman, Thouis R. Jones, and Egon C. Pasztor,
IEEE Computer Graphics and Applications, March/April, 2002
• learn the association between low-resolution
patches and high-resolution patches
• use Markov Network Model (another name
for Markov Random Field) to encourage
adjacent patch coherence
3/7/2003
Super-Resolution
31
Example-based Super-Resolution
William T. Freeman, Thouis R.
Jones, and Egon C.
Pasztor,
IEEE Computer Graphics
and Applications,
March/April, 2002
3/7/2003
Super-Resolution
32
References – “classic”
Irani, M. and Peleg. Improving Resolution by Image Registration. Graphical Models and Image
Processing, 53(3), May 1991, 231-239.
Schultz, R.R.; Stevenson, R.L. Extraction of high-resolution frames from video sequences. IEEE
Trans. Image Proc., 5(6), Jun 1996, 996-1011.
Elad, M.; Feuer, A.. Restoration of a single superresolution image from several blurred, noisy, and
undersampled measured images. IEEE Trans. Image Proc., 6(12) , Dec 1997, 1646-1658.
Elad, M.; Feuer, A.. Super-resolution reconstruction of image sequences. IEEE PAMI 21(9), Sep
1999, 817-834.
Capel, D.; Zisserman, A.. Super-resolution enhancement of text image sequences. CVPR 2000, I600-605 vol. 1.
Chaudhuri, S. (editor). Super-Resolution Imaging. Kluwer Academic Publishers. 2001.
3/7/2003
Super-Resolution
33
References – strong priors
Freeman, W.T.; Pasztor, E.C.. Learning low-level vision, CVPR 1999, 182-1189 vol.2
William T. Freeman, Thouis R. Jones, and Egon C. Pasztor, Example-based super-resolution,
IEEE Computer Graphics and Applications, March/April, 2002
Baker, S.; Kanade, T. Hallucinating faces. Automatic Face Gesture Recognition, 2000, 83-88.
Ce Liu; Heung-Yeung Shum; Chang-Shui Zhang. A two-step approach to hallucinating faces:
global parametric model and local nonparametric model. CVPR 2001. I-192-8.
3/7/2003
Super-Resolution
34