Transcript Computer Vision: Multiview Stereo

CSCE 641 Computer Graphics:
Image-based Modeling
Jinxiang Chai
Image-based modeling
Estimating 3D structure
Estimating motion, e.g., camera motion
Estimating lighting
Estimating surface model
Traditional modeling and rendering
User input
Texture map
survey data
modeling
Geometry
Reflectance
Light source
Camera model
For photorealism:
- Modeling is hard
- Rendering is slow
rendering
Images
Can we model and render this?
What do we want to do for this model?
Image based modeling and rendering
Images
user input range
scans
Image-based
modeling
Model
Image-based
rendering
Images
Spectrum of IBMR
Model
Panoroma
Image
based
modeling
Images
user input range
scans
Images + Depth
Geometry+ Images
Imagebased
rendering
Camera + geometry
Geometry+ Materials
Light field
Kinematics
Dynamics
Etc.
Images
Spectrum of IBMR
Model
Panoroma
Image
based
modeling
Images
user input range
scans
Images + Depth
Geometry+ Images
Imagebased
rendering
Camera + geometry
Geometry+ Materials
Light field
Kinematics
Dynamics
Etc.
Images
Spectrum of IBMR
Model
Panoroma
Image
based
modeling
Images
user input range
scans
Images + Depth
Geometry+ Images
Imagebased
rendering
Camera + geometry
Geometry+ Materials
Light field
Kinematics
Dynamics
Etc.
Images
Stereo reconstruction
Given two or more images of the same scene or object,
compute a representation of its shape
known
camera
viewpoints
What are some possible applications?
3D modeling
From one stereo pair to a 3D head model
[Frederic Deverney, INRIA]
3D modeling
The Digital Michelangelo Project, Levoy et al.
Optical mocap
Vicon mocap system
Z-keying: mix live and synthetic
Takeo Kanade, CMU (Stereo Machine)
Virtualized RealityTM
[Takeo Kanade et al., CMU]
• collect video from 50+ stream
• reconstruct 3D model sequences
•
• steerable version used for
SuperBowl XXV “eye vision”
http://www.cs.cmu.edu/afs/cs/project/VirtualizedR/www/VirtualizedR.html
View interpolation
input
depth image
[Szeliski & Kang ‘95]
novel view
View morphing
Morph between pair of images using epipolar
geometry [Seitz & Dyer, SIGGRAPH’96]
Image warping
Video view interpolation
Performance Interface
Microsoft Natal project
Additional applications?
• Real-time people tracking (systems from Pt. Gray
Research and SRI)
• “Gaze” correction for video conferencing
[Ott,Lewis,Cox InterChi’93]
• Other ideas?
Stereo matching
Given two or more images of the same scene or
object, compute a representation of its shape
What are some possible representations for shapes?
• depth maps
• volumetric models
• 3D surface models
• planar (or offset) layers
Outline
Stereo matching
- Traditional stereo
- Multi-baseline stereo
- Active stereo
Volumetric stereo
- Visual hull
- Voxel coloring
- Space carving
Papers
Stereo matching
•
•
Masatoshi Okutomi and Takeo Kanade. A multiple-baseline stereo. IEEE Trans.
on Pattern Analysis and Machine Intelligence (PAMI), 15(4), 1993, pp. 353--363.
D. Scharstein and R. Szeliski. A taxonomy and evaluation of dense two-frame
stereo correspondence algorithms.
International Journal of Computer Vision, 47(1/2/3):7-42, April-June 2002.
Visual-hull reconstruction
•
•
Szeliski, “Rapid Octree Construction from Image Sequences”, Computer Vision,
Graphics, and Image Processing: Image Understanding, 58(1), 1993, pp. 23-32.
Matusik, Buehler, Raskar, McMillan, and Gortler , “Image-Based Visual Hulls”,
Proc. SIGGRAPH 2000, pp. 369-374.
Photo-hull reconstruction
•
•
Seitz & Dyer, “Photorealistic Scene Reconstruction by Voxel Coloring”, Intl.
Journal of Computer Vision (IJCV), 1999, 35(2), pp. 151-173.
Kutulakos & Seitz, “A Theory of Shape by Space Carving”, International Journal of
Computer Vision, 2000, 38(3), pp. 199-218.
Stereo
scene point
image plane
optical center
Stereo
Basic Principle: Triangulation
• Gives reconstruction as intersection of two rays
• Requires
> calibration
> point correspondence
Camera calibration
From world coordinate to image coordinate
Perspective
projection
Viewport
projection
u
sx
a
u0
v
1
0
0
-sy
0
v0
1
View
transformation
x  w( xs ; p)
2D projections
3D points
Camera parameters
Stereo correspondence
Determine Pixel Correspondence
• Pairs of points that correspond to same scene point
epipolar line
epipolar plane
epipolar line
Epipolar Constraint
• Reduces correspondence problem to 1D search along conjugate
epipolar lines
• Java demo: http://www.ai.sri.com/~luong/research/Meta3DViewer/EpipolarGeo.html
Stereo image rectification
Stereo image rectification
•
•
•

reproject image planes onto a common
plane parallel to the line between optical centers
pixel motion is horizontal after this transformation
two homographies (3x3 transform), one for each
input image reprojection
C. Loop and Z. Zhang. Computing Rectifying Homographies for
Stereo Vision. IEEE Conf. Computer Vision and Pattern
Recognition, 1999.
Rectification
Original image pairs
Rectified image pairs
Stereo matching algorithms
Match Pixels in Conjugate Epipolar Lines
• Assume brightness constancy
• This is a tough problem
• Numerous approaches
> A good survey and evaluation:
http://www.middlebury.edu/stereo/
Your basic stereo algorithm
For each epipolar line
For each pixel in the left image
•
compare with every pixel on same epipolar line in right image
•
pick pixel with minimum matching cost
Improvement: match windows
•
•
This should look familiar..
Can use Lukas-Kanade or discrete search (latter more common)
Window size
W=3
Effect of window size
• Smaller window
+
-
• Larger window
+
-
W = 20
Stereo results
• Data from University of Tsukuba
• Similar results on other images without ground truth
Scene
Ground truth
Results with window search
Window-based matching
(best window size)
Ground truth
Better methods exist...
State of the art method
Boykov et al., Fast Approximate Energy Minimization via Graph Cuts,
International Conference on Computer Vision, September 1999.
Ground truth
Stereo reconstruction pipeline
Steps
•
•
•
•
Calibrate cameras
Rectify images
Compute disparity
Estimate depth
Stereo reconstruction pipeline
Steps
•
•
•
•
Calibrate cameras
Rectify images
Compute disparity
Estimate depth
What will cause errors?
•
•
•
•
•
•
Camera calibration errors
Poor image resolution
Occlusions
Violations of brightness constancy (specular reflections)
Large motions
Low-contrast image regions
Outline
Stereo matching
- Traditional stereo
- Multi-baseline stereo
- Active stereo
Volumetric stereo
- Visual hull
- Voxel coloring
- Space carving
Depth from disparity
disparity map
input image (1 of 2)
[Szeliski & Kang ‘95]
X
z
x’
x
f
C
f
baseline
C’
3D rendering
Choosing the stereo baseline
all of these
points project
to the same
pair of pixels
width of
a pixel
Large Baseline
What’s the optimal baseline?
• Too small: large depth error
• Too large: difficult search problem
Small Baseline
The effect of baseline on depth estimation
1/z
width of
a pixel
pixel matching score
1/z
width of
a pixel
Multi-baseline stereo
Basic Approach
• Choose a reference view
• Use your favorite stereo algorithm BUT
> replace two-view SSD with SSD over all baselines
Limitations
• Must choose a reference view (bad)
• Visibility!
CMU’s 3D Room Video
Outline
Stereo matching
- Traditional stereo
- Multi-baseline stereo
- Active stereo
Volumetric stereo
- Visual hull
- Voxel coloring
- Space carving
Active stereo with structured light
Li Zhang’s one-shot stereo
camera 1
projector
camera 1
projector
camera 2
Project “structured” light patterns onto the object
• simplifies the correspondence problem
Active stereo with structured light
Laser scanning
Digital Michelangelo Project
http://graphics.stanford.edu/projects/mich/
Optical triangulation
• Project a single stripe of laser light
• Scan it across the surface of the object
• This is a very precise version of structured light scanning
Laser scanned models
The Digital Michelangelo Project, Levoy et al.
Laser scanned models
The Digital Michelangelo Project, Levoy et al.
Desktop scanner
Convenient to use
Good quality
Relatively low-cost
- next engine (about 2k)