Transcript Robust Flash Deblurring

Shaojie Zhuo, Dong Guo, Terence Sim
School of Computing, National
University of Singapore
CVPR2010
Key words: image deblur, flash/noflash technique, blur kernel estimate,
sharp image reconstruction
Reporter: 周 澄(A.J.)
01/16/2011
Outline
•Goal
•Major contributions
•Related work
•Flash deblurring framework
‐MAP optimization
•Practical implementation
•Results
•Limitations
•Future works
Goal
•Deblur a shaken image by using a
corresponding flash image generated
from a conventional hand-held camera.
Major contributions
•Propose a novel approach by adding a
flash image as a key constraint.
•Well handle both the flash artifacts
and the deconvolution artifacts.
‐ Additional constraints introduced.
•High quality results.
‐ Insensitive to noise, fine image details.
Related work
•Non-blind image deconvolution
(Single image method).
Def:
Given the estimated blur kernel,
the second step is to reconstruct a
sharp image from the blurred image.
Related work
•[Fergus et al. TOG2006] Removing
camera shake from a single
photograph.
Used a variational Bayes inference
method with natural image statistics
to estimate the motion blur kernel.
Related work
•[Jia. CVPR 2007] Single image motion
deblurring using transparency.
Investigated the relationship between
object boundary transparency and
the image motion blur and
estimated the blur kernel from the
alpha matte of motion objects.
Related work
•[Shan et al. TOG 2008] High-quality
motion deblurring from a single image.
Formulated the deblurring problem
as an MAP problem and proposed a
high-order derivatives image noise
model and a local image prior to
avoid trivial solution.
Related work
•Fergus, Jia, and Shan’s method are all
able to obtain accurate kernels when
the blur is small.
=> Fail at large shaking & noise.
Related work
•Wiener filter and the Richardson-Lucy
(RL) deconvolution.
[Levin et al. CVPR2009]
Understanding and evaluating blind
deconvolution algorithms.
Related work
•Wiener filter and the Richardson-Lucy
(RL) deconvolution.
=> Suffer from deconvolution
artifacts such as amplified noise and
ringing artifacts.
Related work
•Regularization methods.
‐In order to reduce artifacts.
[Wang et al. SIIMS2008] A new
alternating minimization algorithm for
total variation image reconstruction.
[Yuan et al. TOG2008] Progressive
interscale and intra-scale non-blind
image deconvolution.
Related work
•Regularization methods.
=> Lost fine image details since you
cannot separate image details from
artifacts like noise or ringing artifacts
properly.
Related work
•Additional hardware(Hybrid camera).
High resolution camera + low
resolution video camera.
=> Some image details still lost due to
non-invertible motion blur.
Related work
•Multiple images solution.
[Yuan et al. TOG2007] Image deblurring
with blurred/noisy image pairs.
[Chen et al. CVPR2008] Robust dual
motion deblurring.
[YW Tai et al. CVPR2005] Local color
transfer via probabilistic segmentation
by expectation-maximization.
Related work
•Flash/no-flash technique.
[Agrawal et al. TOG2005] Removing
photography artifacts using gradient
projection and flashexposure sampling.
[Petschnigg et al. TOG2004] Digital
photography with flash and no-flash
image pairs.
[Eisemann et al. TOG2004] Flash
photography enhancement via intrinsic
relighting.
Related work
•Traditional flash/no-flash technique.
=> Need good alignment between
two images.
Flash deblurring framework
•Input:
A blur image B and corresponding
flash image F.
•Output:
A visually pleasant sharp image I with
least flash artifacts or deconvolution
artifacts.
Flash deblurring framework
•Problem formulation:
Given the blurred image B and flash
image F, our goal is to estimate a blur
kernel K and a sharp image I, so that
I,K and B can be represented by the
convolution model and the gradients
of I are close to those in F.(We talk
about gradients later)
Flash deblurring framework
Maximum-a-posteriori (MAP) framework
Kernel
estimation
Sharp image
reconstruction
The convolution model
•B = I * K + n,
where n is the image noise which
modeled as a set of independent and
identically distributed(i.i.d.) Gaussian
noise.
MAP optimization
•Problem formulation:
where L(．) = - log (p(．))
MAP optimization
•Likelihood term(SSD):
•Analyze the SSD of the estimated
kernel convolution result and the blur
image.
MAP optimization
•Kernel prior term:
where the parameter α≦1;
α=0.8 used in the paper.
Flash gradient constraint
•Key idea: The robust flash gradient
constraint encourages the gradients of
reconstructed image to be close to
those in F, while at the locations of
flash artifacts, ambient shadows or
noise it allow their gradient to differ to
avoid flash artifacts and keep ambient
illumination.
MAP optimization
•Flash gradient constraint: observation.
1D scanlines of intensities and
gradients in R channel of the
three images.
In the gradient plot, ΔI(cyan)
is much close to ΔF(magenta),
which acts as a guide to
reconstruct the sharp image I.
MAP optimization
•Flash gradient constraint:
where
is the
Lorentzian robust estimator and ε is a
predefined constant.
MAP optimization
•Objective function:
Here we have the likelihood term, flash
gradient constraint and kernel prior
term, where λf and λk are the used to
balance the three terms.
Flash deblurring framework
Maximum-a-posteriori (MAP) framework
Kernel
estimation
Sharp image
reconstruction
Kernel
estimation
Sharp im
reconstru
•Fix K, we can estimate I by solving:
Where the weight of re-weight least
square for each pixel i at each iteration:
Kernel
estimation
Sharp im
reconstru
•Fix I, we can estimate K by solving:
•Both equation can be solved by
iterative re-weight least squares(IRLS).
Kernel
estimation
Sharp im
reconstru
•The two steps are alternated until
diff(K) is smaller than the threshold.
•To avoid local minimum when blur
kernel is large, the kernel estimation is
performed in a coarse-to-fine manner
in the scale space.
ernel
mation
Sharp image
reconstruction
•Three common artifacts.
1. Flash artifact regions.
2. λf is set to be large to suppress
noise or ringing artifacts.
3. Over-saturated regions.
ernel
mation
Sharp image
reconstruction
•Build a mask image M to change the
weight of flash gradient constraint
locally.
ernel
mation
Sharp image
reconstruction
•Flash artifacts detect.
‐|| K * I - F ||2.
‐Only need to manually mark the the
flash shadow edge, since the shadow
regions still contains useful gradients.
ernel
mation
Sharp image
reconstruction
•Add in the sparse gradient constraint.
The objective func becomes:
where “ο” denotes the pixel-wise
multiplication operator.
•Solved by IRLS.
Practical implementation
•Take flash image first, and then use high
speed capturing mode capture blur image.
•As the time between two shots is small,
the the motion during the shots is
basically a translation, which just causes a
shift in the estimated blur kernel.
Therefore, no image alignment is required.
Resaults – kernel RMS error
Resaults
Resaults
Results
Limitations
•It cannot handle the spatially invariant
motion blur model.
‐ Additional alignment needed.
‐ Ex: Busy traffic.
•The exposure time between flash/noflash images should be close.
‐ Temporal incoherence.
•Two images should share same
aperture value.
‐ May generate blur artifacts caused by different focus.
Future works
•Extend to video.
‐ Coherent preserve.
•Support hybrid system such as
combining with IR depth sensor or
stereo sensor to get more information
from image.