Transcript PowerPoint **

Image Matting and Its Applications
Chen-Yu Tseng
Advisor: Sheng-Jyh Wang
2012-10-29
Image Matting
• A process to extract foreground objects from an image, along with an
alpha matte (the opacity of the foreground color)
Input Image
Alpha Matte
Extracted Foreground
Two Approaches of Image Matting
• Supervised Matting
• With User’s Guidance
• Unsupervised Matting
•
Without User’s Guidance
Input Image
User’s Guidance
e.g. Trimap:
White  Foreground
Black  Background
Unknown  Gray
Two Schemes of Supervised Matting
Propagation-based Scheme
Sampling-based Scheme
• Infer Alpha Matte with
Propagation through a
Graphical Model
• Infer Alpha Matte with Some
Color Samples
• A Local-based Approach
• A Global-based Approach
Unknown
Pixel
Foreground Pixel
Unknown Pixel
Background Pixel
Foreground
Color Set
Background
Color Set
Propagation-based scheme Matting Laplacian Approach
• A Graphical Model with Connectivity between Pixels
• The Connectivity Is Learned from the Image Structure
• Capability for Dealing with Both
• Supervised Matting (Inference Problem)
• Unsupervised Matting (Decomposition Problem)
Foreground Pixel
Unknown Pixel
Background Pixel
Reference of Matting Laplacian Approach
• First proposed by Levin et al. for supervised matting (closed-form matting)
• A. Levin, D. Lischinski, Y. Weiss. “A Closed Form Solution to Natural Image
Matting,” IEEE T. PAMI, vol. 30, no. 2, pp. 228-242, Feb. 2008.
• Extended to unsupervised matting (spectral matting)
• A. Levin, A. Rav-Acha, D. Lischinski. “Spectral Matting,” IEEE T. PAMI, vol. 30, no.
10, pp. 1699-1712, Oct. 2008.
• Extended to learning-based matting
• Y. Zheng and C. Kambhamettu. “Learning based digital matting,” In ICCV, pages
889–896, 2009.
• Extended to multi-layer matting
• D. Singaraju, R. Vidal. “Estimation of Alpha Mattes for Multiple Image Layers,”
IEEE T. PAMI, vol. 33, no. 7, pp. 1295-1309, July 2011.
Matting Laplacian
Background
Estimating
Pair-wise
Affinity
Input Image
Supervised
Matting
Graphical Model
Foreground
Node: Image Pixels
Edge: Affinity
Matting Laplacian Matrix:
Recording the Connectivity
between Pair of Pixels
Introduction of Graph Laplacian
Vertex Index
𝑊: Adjacency Matrix
𝑊 = (𝑤𝑖𝑗 )𝑖,𝑗=1,…,𝑛
1
𝐿: Laplacian Matrix
𝐿 =𝐷−𝑊
𝐷 : Degree Matrix
𝑛
𝑑𝑖𝑖 =
𝑤𝑖𝑗
𝑗=1
2
3
4
5
A Graph with Five Vertexes
1
2
3
4
5
1
0
1
1
0
0
2
1
0
1
0
0
3
1
1
0
0
0
4
0
0
0
0
1
5
0
0
0
1
0
𝑊: Adjacency Matrix
Cutting Cost Function with Graph Laplacian
Cost Function for Cutting Criterion
1
𝜶 𝐿𝜶 =
2
𝑛
𝑇
Low-cost
Assignment
𝟏
𝟏
𝜶= 1
0
0
𝑤𝑖𝑗 𝛼𝑖 − 𝛼𝑗
2
𝑖,𝑗=1
1
2
3
4
5
High-cost
Assignment
𝟎
𝟏
𝜶= 0
1
0
1
2
3
4
5
Construction of Matting Laplacian
• Color-model-based Approach (Original)
• Estimating Affinity Based on Relative Color Distance
• Learning-based Approach (Extended)
• Learning Affinity Based on Image Structure
Construction of Matting Laplacian
Color-model-based Approach
b
𝐼𝑖
𝜇𝑘
𝐼𝑗
g
r
Input Image
Color Distribution
A. Levin, D. Lischinski, Y. Weiss. “A Closed Form Solution to Natural
Image Matting,” IEEE T. PAMI, vol. 30, no. 2, pp. 228-242, Feb. 2008.
Construction of Matting Laplacian
Learning-based Approach
• Learning Affinity among Local Pixels
Linear Alpha-color Model
for Single Pixel:
𝛼𝑖 = 𝐱𝑖 𝑇 𝜷 + 𝛽0
= 𝐱𝑖
𝑇
𝜷
1
𝛽0
𝛼𝑖 : Alpha Value for Pixel i
𝐱 𝑖 : Feature Vector ([𝑟𝑖 , 𝑔𝑖 , 𝑏𝑖 ]𝑇 )
𝜷
: Linear Coefficient
𝛽0
Extending to a Local Patch q
Assuming all Pixels Sharing
the Same Linear Coefficient
𝜶𝑞 = 𝐗 𝒒 𝑇
𝜷
𝛽0
𝜶𝑞 : Alpha Vector for Patch q
𝐗 𝑖 : Feature Matrix
𝜷
: Linear Coefficient
𝛽0
Construction of Matting Laplacian
Learning-based Approach
Derived Linear Coefficient
𝜷
𝜷
= arg min 𝜶𝑞 − 𝐗 𝒒 𝑇
𝛽0
𝛽0
𝜷,𝛽0
= 𝐗 𝒒 𝑇 𝐗 𝒒 + 𝜆𝑟 𝐈
−𝟏
2
𝐗 𝒒 𝜶𝑞
Rewritten Linear Model
𝜶𝑞 = 𝐗 𝒒 𝑇
= 𝐗𝒒
𝑇
𝜷
𝛽0
𝑇
𝐗 𝒒 𝐗 𝒒 + 𝜆𝑟 𝐈
−𝟏
𝐗 𝒒 𝜶𝑞
+ 𝜆𝑟 𝜷𝑇 𝜷
Construction of Matting Laplacian
Local Cost Function
Local Linear Model
𝜶𝑞 = 𝐗 𝒒 𝑇 𝐗 𝒒 𝑇 𝐗 𝒒 + 𝜆𝑟 𝐈
−𝟏
𝐗 𝒒 𝜶𝑞
Local Cost Function
E 𝜶𝑞 = 𝜶𝑞 − 𝐗 𝒒
𝑇
𝑇
𝐗 𝒒 𝐗 𝒒 + 𝜆𝑟 𝐈
−𝟏
𝐗 𝒒 𝜶𝑞
2
Patch q
𝑇
= 𝜶𝑞 𝑳𝑞 𝜶𝑞
𝑳𝑞 : Local Laplacian Matrix for Patch q
Input Image
Construction of Matting Laplacian
Local  Global
Local Cost Function
E 𝜶𝑞 = 𝜶𝑞 − 𝐗 𝒒
𝑇
𝑇
𝐗 𝒒 𝐗 𝒒 + 𝜆𝑟 𝐈
−𝟏
𝐗 𝒒 𝜶𝑞
2
𝑇
= 𝜶𝑞 𝑳𝑞 𝜶𝑞
𝑳𝑞 : Local Laplacian Matrix for Patch q
Patch q
Global Cost Function
𝐽 𝜶 =
𝜶𝑞 𝑇 𝑳𝑞 𝜶𝑞
E 𝜶𝑞 =
𝑞
= 𝜶𝑇 𝑳𝜶
𝑞
Input Image
Supervised Matting (Closed-form Matting)
Input Image
Foreground Pixel
Unknown Pixel
Background Pixel
User’s Guidance, 𝜷
Cost Function for Supervised Matting
Optimal Solution
𝜶∗ = (𝑳 + 𝚲)−1 𝚲𝜷
𝐸 𝜶 = 𝜶𝑇 𝑳𝜶 + (𝜶 − 𝜷)𝑇 𝚲(𝜶 − 𝜷)
Affinity Cost
Data Cost
Foreground
Background
Unknown
𝜷 𝒊
1
0
-
𝚲 𝒊, 𝒊
1
1
0
Experimental Results
Input Image
Alpha Matte
Synthesized Result
Unsupervised Matting (Spectral Matting)
• Solving Alpha Matte without User’s Guidance
• Procedures
• Decomposing Image into Several Matting Components
• Combining Matting Components into Alpha Matte
Spectral Clustering
arg min𝑓 𝒇𝑇 𝐿𝒇
s.t. 𝒇𝑇 𝒇=1
𝐿𝒇 = λ𝒇
𝒇: Eigenvector
λ: Eigenvalue
1. L is symmetric and positive semi-definite.
2. The smallest eigenvalue of L is 0, the corresponding
eigenvector is the constant one vector 1.
3. L has n non-negative, real-valued eigenvalues
0= λ 1 ≦ λ 2 ≦ . . . ≦ λ n.
1
2
3
4
5
A Graph Example
1
2
3
4
5
1
2
-1
-1
0
0
0.047
0.577
2
-1
2
-1
0
0
0.047
0.577
3
-1
-1
2
0
0
0.047
0.577
4
0
0
0
1
-1
0.047
0
5
0
0
0
-1
1
0.047
0
𝐿: Laplacian Matrix
𝒇1
𝒇2
Spectral Clustering & Matting Components
2
-1 -1 0
-1 2
-1 0
0
0
0
1
0
0
0
0
0
1
0
0
-1 -1 2
0
0
0
0
1
0
0
0
0
0
1
1
0
0
0
1
0
0
0
0
-1 -1 0
0
0
1
0
0
0
0
0
0
1
1
0
0
1
0
0
0
0
0
-1 -1
0
0
1
𝐿: Laplacian Matrix
Zero-Eigenvectors
× 𝑹3×3
Linear
Transformation
Binary Indicating Vectors
Overview of Spectral Matting
K-means
Clustering
&
Linear
Transformation
Input Image
Matting
Laplacian
Smallest Eigenvectors
Matting Components
Spectral Clustering & K-means
Pixel i
Input Image
s-dimensional
Space
…
K-means
Clustering
s-smallest
Eigenvectors
Generating Matting Components
Smallest Eigenvectors
𝑬 = 𝒆𝟏
… 𝒆𝒔
K-means
𝒎𝒌
..
…
…
…
.
Projection into Eigen Space
𝜶𝒌 = 𝑬𝑬𝑻 𝒎𝒌
Reconstructing Alpha Matte from
Matting Components
Input Image
Matting Components
+
+
Selected Matting Components
=
Alpha Matte
Reconstructing Alpha Matte by
Grouping Matting Components
Alpha Matte Generation
𝜶 = 𝜶1
… 𝜶𝑘 𝒃
𝒃: Combination Vector
Matting cost function
𝐽 𝜶 = 𝜶𝑇 𝑳𝜶
= 𝒃𝑇 𝜶1
… 𝜶𝑘 𝑇 𝑳 𝜶1
… 𝜶𝑘 𝒃
= 𝒃𝑇 𝜱𝒃
Evaluating All Grouping Hypothesis to Derive the Optimal Alpha Matte
Results by Levin et al.
Summary
• Constructing Matting Laplacian
• Solving Supervised Matting Problem
• Solving Unsupervised Matting Problem
Proposed Approaches
• Efficient Cell-based Framework for Reducing Computations
• Multi-scale Analysis
• Extended Applications (Depth Image Reconstruction)
Depth Reconstruction
in Shape From Focus (SFF)
Input Image
Reconstructed
Depth
Depth Reconstruction from Single Image
Input Image
Reconstructed
Depth
Cell-based Framework
Pixel-wise
Affinity
Conventional
Matting Laplacian
Pixel-wise Data Distribution
Cell-wise
Affinity
Image
Cell-wise Data Distribution
Cell-based
Matting Laplacian
Multi-scale Affinity Learning
Image &
Computation
Patterns
Pixel-based
Approach
Cell-based
Approach
Multi-scale Affinity Learning
…
Finest Level
…
Cell-based Graph
Coarsest Level
Results of Reconstructed Alpha Matte
Input
(a) Grouping Results by
Levin et al.
(b) Grouping Results by
Levin et al. with
Coarse-to-fine Scheme.
(c) Ours
1st Rank
2nd Rank
Results
(a) Input images (b) Levin’s result
(c) Our result
Proposed Approaches
• Efficient Cell-based Framework for Reducing Computations
• Multi-scale Analysis
• Extended Applications (Depth Image Reconstruction)
Depth Reconstruction
in Shape From Focus (SFF)
Input Image
Reconstructed
Depth
Depth Reconstruction from Single Image
Input Image
Reconstructed
Depth
Depth Reconstruction in Shape From Focus (SFF)
Optical
Direction
W2
W1
W2
Optical
Direction
Multi-focus Image Sequence
W1
Focus
Value
Low-SNR Problem
• Spatially Varying Precision
• Low-texture  Low-SNR
• Leading Noisy Result
Lowprecision
Highprecision
Input Image
Observation
Proposed Maximum-a-posteriori Estimation
Local
Learning
Learning-based Graph
Inference
Multi-focus Image Sequence
Reconstructed Depth
Proposed Maximum-a-posteriori Estimation
𝐷 ∗ : Optimal Result
𝐷: Depth Image
𝑌: Observation
𝐼: Input Image
𝐷∗ = max 𝑝 𝐷 𝑌, 𝐼
𝑝 𝐷 𝑌, 𝐼 ∝ 𝑝 𝑌 𝐷, 𝐼 𝑝 𝐷 𝐼
Posterior
Likelihood
Prior
Learned from Image
Local Observation
with Spatial-varying Precision
Likelihood Model
𝑝 𝐷 𝑌, 𝐼 ∝ 𝑝 𝑌 𝐷, 𝐼 𝑝 𝐷 𝐼
Posterior
Local Observation
with Spatial-varying Precision
− log 𝑝 𝑌 𝐷, 𝐼
= (𝒅 − 𝒚)𝑇 𝚲(𝒅 − 𝒚)
Likelihood
Prior
Lowprecision
Highprecision
Input 𝑰
Precision 𝚲
Observation 𝒀
Result 𝐷 ∗
𝑝 𝐷 𝑌, 𝐼 ∝ 𝑝 𝑌 𝐷, 𝐼 𝑝 𝐷 𝐼
Prior Model
Posterior
Likelihood
Learning from Input Image
− log 𝑝 𝐷 𝐼 = 𝒅𝑇 𝑳𝒅
Local
Learning
Learning-based Graph
Multi-focus Image Sequence
Prior
Maximum-a-posteriori Estimation for
Depth Reconstruction
𝐷∗ = max 𝑝 𝐷 𝑌, 𝐼
𝑇
− log 𝑝 𝐷 𝑌, 𝐼 ∝ 𝒅 − 𝒚 𝚲 𝒅 − 𝒚 + 𝒅𝑇 𝑳𝒅
𝐷∗ = 𝑳 + 𝚲
Input Image
−1
𝚲𝒅
Observation
Reconstructed Depth
Results of Shape from Focus
Input Image
S. Nayar, 1994
M. Mahmood, 2012
T. Aydin, 2008
Ours
Conclusions
• Construction of Matting Laplacian
• Conventional Approach
• Multi-scale Cell-based Approach
• Supervised Matting
• Spectral Matting
• Depth Reconstruction