Transcript Defocus Magnification Soonmin Bae and Frédo Durand MIT CSAIL 1

Defocus Magnification
Soonmin Bae and Frédo Durand
MIT CSAIL
1
SLR vs. Point-and-Shoot
SLR camera
Point-and-Shoot camera 2
Shallow Depth of Field
Sharp foreground with blurred background
3
A Point-and-Shoot Camera
Background is not blurred enough
4
Defocus
• Point in focus: rays converge to sensor
• Farther points are blurrier
focal plane
lens
sensor
5
Defocus and Aperture size
• Bigger aperture produces more defocus
- F-number N gives the aperture diameter A as a fraction of
the focal length f (A = Nf )
- Example : f = 100 mm, f/2 A = 50mm, f/4 A = 25mm
f/4
f/2
6
focal plane
lens
sensor
Defocus and Aperture size
• Aperture : the size of the lens opening
- Wide aperture : shallow depth of field
- Pinhole aperture : infinite depth of field
Wide aperture (f/2)
blurred background
Narrow aperture (f/32)
7
background remained sharp
Defocus and Sensor size
• Sensor size
- Small sensor  small lens  less defocus
- Defocus size is mostly proportional to the sensor size
(see paper)
Large sensor (22.2 x 14.8), f/2.8 Small sensor (7.18 x 5.32), f/2.8
8
blurred background
background remained sharp
Defocus and Sensor size
• Sensor size
- Small sensor  small lens  less defocus
- Defocus size is mostly proportional to the sensor size
(see paper)
Large sensor (22.2 x 14.8), f/2.8 Small sensor (7.18 x 5.32), f/2.8
9
blurred background
background remained sharp
Goals
• Magnify defocus given a single image
- Blur blurry regions and keep sharp regions sharp
• Simulate shallow depth of field
input
magnified defocusing result
10
Overview
1. User provides a single input photograph
2. Our system automatically produces the defocus map
3. We use Photoshop’s lens blur to generate the defocus
magnified result
input
our defocus map
11
Increased defocus
We do not require …
precise depth estimation
disambiguation b/w out-of-focus edges and
originally smooth edges
• But we simply compute the amount of blur and
increase it
12
Related Work – Depth from De/focus
• Seeks the exact depth map
[Horn 68; Pentland 87; Darrell 88; Ens 93; Nayar 94; Watanabe 98; Favaro 02; Jin 02;
Favaro 05; Hasinoff 06]
- is a hard problem
- needs multiple images with different focus settings
near-focused
far-focused
depth from defocus
[Durand 02]
[Watanabe 98]
In contrast,
we want to estimate the blur kernel, not the depth
In addition, we analyze the blur kernel only at edges
13
Related Work
- remove blur from images
• Blind deconvolution [Reeves 92; Trussell 92; Özkan 94, Fergus 06]
• Extended depth of field using multiple images
[Adelson 83; Eltoukhy 03; Agarwala 04; Kubota 05]
near-focused
far-focused
all-in-focus result
In contrast, we want to increase defocus
[Kubota 98]
In addition, we use a single image to estimate spatially variant
14
blurs
Related Work
- computational camera
• Defocus manipulation
- [Ng 05, Green 07, Levin 07, Moreno-Noguer 07, Veeraraghavan 07]
[Levin 07]
In contrast, we want to use image-processing techniques
15
Related Work – Synthetic Lens Blur
• Given image & depth map,
simulate defocus
[Potmesil 81]
- e.g. Adobe Photoshop and
Depth of Field Generator Pro
We use Photoshop Lens Blur to generate results with
our defocus map instead of a depth map
16
Our work
• Measure blurriness
- Estimate the spatially-varying amount of blur at edges
• Propagate blurriness (defocus map)
- Assume that blurriness is smooth except at image edges
• Blur the blurry regions
- Use Photoshop lens blur
input
blur (defocus)
measure
defocus map
result
17
Our work
• Measure blurriness
- Estimate the spatially-varying amount of blur at edges
• Propagate blurriness (defocus map)
- Assume that blurriness is smooth except at image edges
• Blur the blurry regions
- Use Photoshop lens blur
input
blur (defocus)
measure
defocus map
result
18
Blur Estimation at Edges
• an edge
- a step function in intensity
• the blur of this edge
- a Gaussian blurring kernel
edge
gaussian blur
blurred edge
19
Blur Estimation at Edges
• an edge
- a step function in intensity
• the blur of this edge
- a Gaussian blurring kernel
edge
gaussian blur
blurred edge
• Multiscale edge detector
• Blur estimation at edges
20
Blur Estimation at Edges [Elder 98]
• Multiscale edge detector
- output : a sparse set of pixels
• Blur amount σb at edges
- related to the distance between extrema of the second derivative
- Elder and Zucker use the zero-cossing of the third derivative
d
21
edge
gradient
2nd
derivative
- works in simple
cases (i.e. a single
line, no texture)
Y axis
Distance between Zero-crossing of
the Third Derivative [Elder 98]
input
Y axis
- sensitive to the
influence of scene
events nearby
input
zero-crossing of
22
the third derivative
Blur Estimation at Edges
• Fit response models of various sizes
less blurry
d
edge
2nd derivative
more blurry
response model
23
Robust Blur Estimation
• Successfully measure the blur size in spite of the
influence of scene events nearby
blurry
sharp
input
our blur measure
24
Our blur measure
• A sparse set
- values only at edges
- Grey means no value
blurry
input
blur measure
sharp
25
Refinement of Blur Estimation
• Erroneous blur estimates
- due to soft shadows and glossy highlights
blurry
input
blur measure
sharp
26
Refinement of Blur Estimation
• Erroneous blur estimates
- due to soft shadows and glossy highlights
blurry
input
blur measure
sharp
27
Remove Outliers
• Using cross bilateral filtering [Eisemann 04, Petschnigg 04]
- a weighted mean of neighboring blur measures
- Smoothes the blur measure near in spatial distance and close in
range difference of a reference image
blurry
before refinement
after refinement
sharp
28
Our work
• Measure blurriness
- Estimate the spatially-varying amount of blur at edges
• Propagate blurriness (defocus map)
- Assume that blurriness is smooth except at image edges
• Blur the blurry regions
- Use Photoshop lens blur
input
blur (defocus)
measure
defocus map
result
29
Blur Propagation
• Given a sparse set of the blur measure (BM)
• Propagate the blur measure to the entire image
- Assumption : blurriness (B) is smooth except at image edges
- Inspired by [Levin et al. 2004]
input
blur measure
30
Blur Propagation
• Given a sparse set of the blur measure (BM)
• Propagate the blur measure to the entire image
- Assumption : blurriness (B) is smooth except at image edges
• We minimize
data term
smoothness term
proporsional to
e -|| C(p) – C(q) ||
2
31
Blur Propagation
• Edge-preserving propagation
- propagation stops at input edges
input
blur
defocus
measure
map
32
Our work
• Measure blurriness
- Estimate the spatially-varying amount of blur at edges
• Propagate blurriness (defocus map)
- Assume that blurriness is smooth except at image edges
• Blur the blurry regions
- Use Photoshop lens blur
input
blur (defocus)
measure
defocus map
result
33
Recap
1. User provides a single input photograph
2. Our system automatically produces the defocus map
3. We use Photoshop’s lens blur to generate the defocus
magnified result
input
our defocus map
34
Increased defocus
Input
Defocus Map
Result
35
Input
Result
36
Input
Defocus Map
Result
37
Input
Result
38
Input
Defocus Map
Result
39
Input
Result
40
Comparison with the ground truth
Input (f/8)
our result
ground truth (f/4)
41
Summary
• Analyze existing defocus
- multiscale edge detector & fitting
- non-homogeneous propagation
• Magnify defocus
Future work
• Occlusion boundary
• Video inputs (motion blur)
• Refocusing
42
Preliminary Refocusing Result
• Synthesize refocusing effects
- Perform deconvolution using our defocus map
Input : defocus
blurred fgmap
& sharp bg
Refocusing result
43
Contributions
• Our defocus map captures blurriness
• Our defocus map can be used to increase defocus
44
Acknowledgement
• MIT Computer Graphics Group
• anonymous reviewers
• NSF/ Royal Dutch / Shell Group
• a Microsoft Research New Faculty Fellowship, a
Sloan Fellowship, and a Jamieson chair
• Samsung Scholarship
45