Transcript Depth image generated from the MV

Xu Xuyuan(Evan)
2010/5/20


Most of the TV manufacturers release the
3DTV sets.
Sony 3DTV using Shutter glass technology



Large number of monoscopic videos existing
in many databases
Stereo video market demand
Enable to capture 3D using the 2D camera

3D shutter glass display system

2D-3D generation process
◦ Depth map generation
◦ Depth Image Based rendering

Conclusion

Recommendation

Device
◦ A monitor with frequency 120Hz or above
◦ A shutter glasses

Methodology
◦ Used the monitor to shift the image concurrently.
DIBR --Depth Image Based Rendering
Input
Video
3D Image
Warping
Hole filling
Depth map generation
Motion
estimation
Filtering
Color
Segmentation
Fusion
Modification of
DIBR
Output 3D
video
Input
Video
Depth map generation
Motion
estimation

Depth Cue -- Motion Parallax
◦ Near objects move faster than the distant objects

Requirement of source video
◦ Have movement

Depth map
◦ A grey scale image
◦ Each pixel represent by 8 bits
Searching Frame
Reference Frame


Used Y (luminance) value for the motion vector
search
sum of absolute difference
◦ (SAD)

Four kind of search:
◦
◦
◦
◦
Full Search
Spiral Search
Diamond Search
Hexagonal search
◦
◦
◦
◦
D(i,j) is the depth value for pixel (i,j)
MV(i,j)x is the X motion vectors values
MV(i,j)y is the Y motion vectors values
C is a custom define scale parameter
Spiral search
Input Image of size: 416*240
Searching method
Time consumption
Full search
12.215 s
Spiral search
12.184 s
Hexagonal search
0.359 s
Diamond search
0.172 s
Hexagonal search
Full Search
Diamond search
Input
Video
Depth map generation
Motion
estimation
Color
Segmentation

1. Step color quantization
◦ Set a quantization step.

2. Color merging
◦ Find out the best two merging color in the image
and merge them into one.
◦ Merging ColorC = |colorA*numA+colorB*numB|/(numA+numB)
◦ Error = (|colorA- colorC|*numA+ |colorB- colorC|*numB)/(numA+numB)
Node Structure
prev
next
Color – R, G B
Number of pixels
Before Merging
q
i
p
j
m
k
p
j
m
After Merging
q
i and j are merging into k (i is updated into k and j is delete)
While (has unassigned value for the segmentation
map)
{
index = Get next unassigned index point
from the segmentation map.
Find the continue area begin with the index in
sMap with the value equal to qMap[index]
Assign a defined segmentation value to sMap
}
qMap: quantization map
sMap: segmentation map
Input
Video
Depth map generation
Motion
estimation
Color
Segmentation
Fusion
While (has unassigned value for final depth map)
{
index = Get next unassigned index point from final depth map
Find out the corresponding area in segmentation map and decide
the depth value from average, median and mode.
If number of the frequency depth value from mode is greater than
half
Final depth value = value from mode
else {
If there exists at least two depth value from three fusion
method are similar
Final depth value = interpolation of these depth value
else
Final depth value = value from average fusion
}
}

Final depth map generate from
◦ Depth map from motion estimation
◦ Color segmentation
Fusion
Input
Video
Depth map generation
Motion
estimation
Filtering
Color
Segmentation
Fusion

Gaussian filtering
DIBR
Input
Video
3D Image
Warping
Depth map generation
Motion
estimation
Filtering
Color
Segmentation
Fusion

Formula:
xl  xc 
tc f
h
2Z ( xc , y)
xr  xc 
tc f
h
2Z ( xc , y)
h
tc f
2Z c
DIBR
Input
Video
3D Image
Warping
Hole filling
Depth map generation
Motion
estimation
Filtering
Color
Segmentation
Fusion
Objects
Zdetph
Intersection
Screen
Left eye
Right eye


Why hole filling?
Linear Interpolation to fill the hole.
DIBR
Input
Video
3D Image
Warping
Hole filling
Depth map generation
Motion
estimation
Filtering
Color
Segmentation
Fusion
Modification of
DIBR
Screen
New depth
Zdetph
=255
Depth
Screen
Depth(i,j)=255-Depth(i,j).
Left eye
Right eye
DIBR --Depth Image Based Rendering
Input
Video
3D Image
Warping
Hole filling
Depth map generation
Motion
estimation
Filtering
Color
Segmentation
Fusion
Modification of
DIBR
Output 3D
video
DIBR
Input
Video
3D Image
Warping
Hole filling
Depth map generation
Motion
estimation
Color
Segmentation
Filtering
Fusion
Depth Map
Modification of
DIBR
Output 3D
video




Depth from motion can achieve really time
generation.
Achieve a good 3D effect for the source video
with static background
The output video compatible for most of 3D
display system
GUI – More user friendly interface


Camera motion correction
Multi-view 3D generation
Thank You!