Transcript IBM / IBR “The study of image-based modeling sampled representations of geometry.”

IBM / IBR
“The study of image-based modeling
and rendering is the study of
sampled representations of geometry.”
Marc Levoy
Shortcutting the
vision/graphics pipeline
real world
vision
pipeline
geometry
image-based
rendering
graphics
pipeline
views
(from M. Cohen)
Marc Levoy
Apple QuickTime VR
[Chen, Siggraph ’95]
• outward-looking
– panoramic views taken at regularly spaced points
• inward-looking
– views taken at points on the surface of a sphere
Marc Levoy
View interpolation
from a single view
1. Render object
2. Convert Z-buffer to range image
3. Re-render from new viewpoint
4. Use depths to resolve overlaps
Q. How to fill in holes?
Marc Levoy
View interpolation
from multiple views
1. Render object from multiple viewpoints
2. Convert Z-buffers to range images
3. Re-render from new viewpoint
4. Use depths to resolve overlaps
5. Use multiple views to fill in holes
Marc Levoy
Post-rendering 3D warping
[Mark et al., I3D97]
• render at low frame rate
• interpolate to real-time frame rate
–
–
–
–
interpolate observer viewpoint using B-Spline
convert reference images to polygon meshes
warp meshes to interpolated viewpoint
composite by Z-buffer comparison and conditional write
Marc Levoy
Results
• rendered at 5 fps, interpolated to 30 fps
• live system requires reliable motion prediction
– tradeoff between accuracy and latency
• fails on specular objects
Marc Levoy
Light field rendering
[Levoy & Hanrahan, Siggraph ’96]
• must stay outside convex hull of the object
• like rebinning in computed tomography
Marc Levoy
A light field is an array of images
Marc Levoy
Stanford multi-camera array
(Horowitz, Levoy, Hanrahan)
• cheap imagers + cheap optics +
fast networking + plentiful computation =
high-performance imaging
using an array of low-cost cameras
Marc Levoy
Applications for the array
• key issue is the spacing and
arrangement of the cameras
Marc Levoy
Cameras tightly packed:
high-X imaging
• high-resolution
– by abutting the cameras’ fields of view
• high speed
– by staggering their triggering times
• high dynamic range
– mosaic of shutter speeds, apertures, density filters
• high depth of field
– mosaic of differently focused lenses
Marc Levoy
Cameras widely spaced:
a video-rate light field camera
• compression of (video) light fields
– key is estimation of rough geometry
• autostereoscopic display of light fields
– lens arrays, bumpy mirrored sheets
• shape from light fields
– Do more images help?
Marc Levoy