Transcript IMI PhD Seminar 2014_01_14

Real-time Foreground Extraction
with RGBD Camera for 3D
Telepresence
Presenter: ZHAO Mengyao, PhD, SCE
Supervisor: Asst/P FU Chi-Wing, Philip
Co-Supervisor: A/P CAI Jianfei
Outline
•
•
•
•
•
•
•
Motivation
Related Work
Challenges
Our Approach
Results
Limitation
Future Work
3D Telepresence
3D Telepresence
BeingThere Centre’s RBT Project
IBM’s Holographic 3D cell phone
Foreground Extraction
Related Work
• Chroma Key [1, 2]
Related Work
• Interactive Approach [3-8]
Related Work
Microsoft Kinect [9]
PrimeSense Carmine 3D Sensor [10]
Related Work
• Real-time Foreground Extraction with RGBD
camera:
– FreeCam [11]
Challenges
• Convenience
– Arbitrary background
• High quality
– Natural & smooth boundary
• Automation
– No manual markup
• Real-time
– Support teleconference/telepresence
• Temporal coherency
– Free of flickering artifact
Challenges
• Inaccurate depth map
Red:
Color
Green: Depth
Depth on Color
•Noisy depth map
•Depth/Color not well aligned
•Depth/Color not synchronized
Our Approach
• We aim to
– Perform high-quality coherent foreground extraction in real-time
that could support teleconference and telepresence
• We propose
– An integrated pipeline for robust foreground extraction with
RGBD camera
– A temporal coherent matting approach
– A CUDA based GPU implementation of our approach that
achieves real-time performance
Our Approach –
Matting
where i is the index of
pixel, I is the intensity,
α is the alpha, F is the
foreground, B is the
background
Input
Trimap
Alpha
Our Approach –
Workflow
Our Approach –
Pipeline
Our Approach –
Pipeline
Our Approach –
Temporal Hole Filling with Depth Map Shadow Detection
NMD: no-measured depth [12]
Black: NMD regions
Yellow: mirror-like regions
Raw depth map
Green: out-of-range regions
Red:
shadow regions
Detected shadow Types of NMD regions
Our Approach –
Temporal Hole Filling with Depth Map Shadow Detection
• Temporal Hole-filling:
– For shadow region, apply below:
– For other NMD region, apply joint-bilateral filter
Our Approach –
Pipeline
Our Approach –
Adaptive Binary Mask Generation
color mask
final mask
depth mask
Our Approach –
Pipeline
Our Approach –
Non-local Temporal Matting: intro of closed-form matting
Closed-form matting:
Assumption:
Both F and B are approximately constant over a small window around each pixel.
1
2
Alpha can be obtained by solving:
(3)
Our Approach –
Non-local Temporal Matting: extension of closed-form matting
Assumption: F and B are smooth in local window
Temporal coherency
Assumption: F and B are smooth in both spatial and temporal domain
Our Approach –
Non-local Temporal Matting: 3d non-local neighbor
It+1
It0
It-1
It0
2D neighbor
3D non-local neighbor
is the neighbor of
(4)
(5)
(3)
Our Approach –
Non-local Temporal Matting: volume partition
• The linear equation system is too large
• Use kd-tree segmentation to partition the volumes
• Recursive until number of unknown within each block is smaller than
a threshold
Our Approach –
GPU implementation: CUDA and CULA Sparse
Quantitative Performance
Stage
Average Time
(ms/frame)
FPS
(frame/s)
Background Modeling
2.598231
384.877308
Preprocessing
0.671895
1488.328257
Trimap Generation
3.323211
300.913784
Temporal Matting
48.483765
20.625460
Total
52.478870
19.055288
Results
Comparison with three other state-of-the-art works
Limitation
• Shadow-like region when moving fast
– Reason 1: color/depth not well aligned
– Reason 2: color/depth not synchronized
• Sometimes inaccurate when
foreground/background share similar color
Future Work
• Refine the depth map using attained alpha map
to achieve better 3D representation