Transcript 3D Photography (Image-based Model Acquisition) Funky Image Goes Here 5/1/2000 Deepak Bandyopadhyay / UNC Chapel Hill.

3D Photography
(Image-based Model Acquisition)
Funky Image Goes Here
5/1/2000
Deepak Bandyopadhyay / UNC
Chapel Hill
1
“Analog” 3D photography !
• “3D stereoscopic imaging”
– been around as long as cameras have
– Use camera with 2 or more lenses (or stereo attachment)
– Use stereo viewer to create impression of 3D
11/6/2000
Deepak Bandyopadhyay / 258 / 3D Photography
2
Motivation
• Digitizing real world objects
• Getting realistic models
places
objects
humans
11/6/2000
Deepak Bandyopadhyay / 258 / 3D Photography
3
3D Photography : Definition
• Sometimes called “3D Scanning”
• Use cameras and light to capture the shape &
appearance of real objects
• Shape == geometry (point sampling + surface
reconstruction + fairing)
• Appearance == surface attributes (color/texture,
material properties, reflectance)
• Final result = richly detailed model
11/6/2000
Deepak Bandyopadhyay / 258 / 3D Photography
4
Applications in Industry
• Human body / head / face scans
– Avatar creation for virtual worlds
– 3d conferencing
– medical applications
– product design
– Platforms:
• Cyberware RD3030
• Others (Geomagic, Metacreations, Cyrax, Geometrix…)
11/6/2000
Deepak Bandyopadhyay / 258 / 3D Photography
5
More applications
• Historical preservation, dissemination of museum
artifacts (Digital Michelangelo, Monticello, …)
• CAD/CAM (eg. Legacy motorcycle parts scanned
by Geomagic for Harley-Davidson).
• Marketing (models of products on the web)
• 3D games & simulation
• Reverse engineering
11/6/2000
Deepak Bandyopadhyay / 258 / 3D Photography
6
Technology Overview
• The Imaging Pipeline
–
–
–
–
–
Real World
Optics
Recorder
Digitizer
Vision & Graphics
11/6/2000
Deepak Bandyopadhyay / 258 / 3D Photography
7
Quick Notes on Optics
• Model lenses with all their properties aberration, distortion, flare, vignetting etc.
• We correct for some of these effects (eg.
distortion) in the calibration, ignore others.
• CCD (charged coupled devices) are the
most popular recording media.
11/6/2000
Deepak Bandyopadhyay / 258 / 3D Photography
8
Theory : Passive Methods
•
•
•
•
Stereo pair matching
Structure from motion
Shape from shading
Photometric stereo
11/6/2000
Deepak Bandyopadhyay / 258 / 3D Photography
9
Stereo Matching
• Stereo Matching Basics
– Needs two images, like stereoscopy
– Given correspondence between
points in 2 views, we can find
depth by triangulation
– But correspondence is hard prob!
– A lot of literature on solving it…
• Stereo Matching output
• 3D point cloud
• Remove outliers and pass through surface reconstructor
11/6/2000
Deepak Bandyopadhyay / 258 / 3D Photography
10
Structure from Motion
• Camera moving, objects static
• Compute camera motion and object geometry from
motion of image points
• Assumption - orthographic projn (use telephoto)
• If: world origin = 3D centroid
camera origin = 2D centroid
Then: camera translation drops out
11/6/2000
Deepak Bandyopadhyay / 258 / 3D Photography
11
Structure from Motion
• Camera moving, objects static
• Compute camera motion and object geometry from
motion of image points
11/6/2000
Deepak Bandyopadhyay / 258 / 3D Photography
12
Structure from Motion
• Factorization [Tomasi & Kanade, 92]
• Find M, S using Singular Value Decomposition of
W.
SVD gives:
S’S modulo linear transform A.
Solve for A using constraints on M.
11/6/2000
Deepak Bandyopadhyay / 258 / 3D Photography
13
More methods
• Shape from shading, [Horn]
– Invert Lambert’s Law (L=I k cos )
knowing the intensity at image point
to solve for normal
• Photometric stereo [Woodham]
–
–
–
–
11/6/2000
An extension of the above
Two or more images under different illumination conditions.
Each image provides one normal
Three images provide unique solution for a pixel.
Deepak Bandyopadhyay / 258 / 3D Photography
14
Active Sensing
• Passive methods (eg. stereo matching) suffer from
ambiguities - many similar regions in an image
correspond to a point in the other.
• Project known / regular pattern (“structured light”)
into scene to disambiguate
• get precise reconstruction by combining views
– Laser rangefinder
– Projectors and imperceptible structured light
11/6/2000
Deepak Bandyopadhyay / 258 / 3D Photography
15
Desktop 3D Photography
Jean-Yves Bouguet, Pietro Perona
• An active sensing technique using “weak
structured lighting”
• Need: camera, lamp, chessboard, pencil, stick
• Idea:
– Light object with lamp & aim camera at it
– Move stick around & capture shadow sequence
– Use image of deformed shadow to calc 3D shape
11/6/2000
Deepak Bandyopadhyay / 258 / 3D Photography
16
Desktop 3D Photography
Jean-Yves Bouguet, Pietro Perona
• Computation of 3d position from the plane of
light source, stick and shadow
11/6/2000
Deepak Bandyopadhyay / 258 / 3D Photography
17
Volumetric Methods
Chevette Project, Debevec, 1991
11/6/2000
Deepak Bandyopadhyay / 258 / 3D Photography
18
Voxel Models from Images
• When there are 2 colors in the image - use volume
intersection [Szeliski 1993]
– Back-project silhouettes from camera views &
intersect
11/6/2000
Deepak Bandyopadhyay / 258 / 3D Photography
19
Voxel Models from Images
• With more colors but constrained viewpoints, we
use voxel coloring [Seitz & Dyer, 1997]
– Choose a voxel & project to it from all views
– Color if enough matches
– Prob - determining visibility
of a point from a view
– Solution - depth ordered
traversal using a “view indep.
d.o.” (dist from separating plane)
11/6/2000
Deepak Bandyopadhyay / 258 / 3D Photography
20
Voxel Models from Images
• A view-independent depth order may not exist
(for some configuration of viewpoints / scene geometry).
• Use Space Carving [Kutulakos & Seitz, 1998]
– Computes 3D (voxel) shape from multiple color photos
– Computes “maximally photo-consistent shape”
• maximal superset of all 3D shapes that produce the given photos
11/6/2000
Deepak Bandyopadhyay / 258 / 3D Photography
21
Space Carving
• Algorithm:
a) Initialize V to volume
containing true scene
b) For each voxel,
• check if photo-consistent
• if not, remove (“carve”) it.
• Can be shown to converge to maximal photo-consistent
scene (union of all photo-consistent scenes).
11/6/2000
Deepak Bandyopadhyay / 258 / 3D Photography
22
Space Carving : Results
• House walkthru - 24 rendered input views
• Results best as seen from one of the original views
11/6/2000
Deepak Bandyopadhyay / 258 / 3D Photography
23
Modeling from a single view
(Criminisi et al, 1999)
• Compute 3D affine measurements of the scene
from single perspective image
• Use minimal geom info
– vanishing line for a pencil of
planes || to reference plane
– vanishing point of parallel
lines along a direction
outside reference plane
11/6/2000
Deepak Bandyopadhyay / 258 / 3D Photography
24
Modeling from a single view
(Criminisi et al, 1999)
• Compute “ratio of parallel distances”
• Creating a 3D model from a photograph
– horizontal lines used to compute vanishing line
– parallel vertical lines used to compute vanishing point
• Can generate geometrically correct model from a
Renaissance painting (with correct perspective)
11/6/2000
Deepak Bandyopadhyay / 258 / 3D Photography
25
Extracting color, reflectance
• Photographs have lighting/shading effects that we
estimate (reflectance function) and compensate for
(specular highlight removal) or change (relighting)
• Work of Paul Debevec & others at Berkeley
(acquiring reflectance field)
• Wood et al at U. Washington (surface light lield
for 3D photography)
11/6/2000
Deepak Bandyopadhyay / 258 / 3D Photography
26
Surface Light Field
[Wood et al, 2000]
• A 4D function on the surface - at surface
parameter (u,v), for every direction (,), stores
the color.
• Fixed illumination conditions.
• Photographs taken from a lot of different
directions sample the surface light field.
• Continuous function (piecewise linear over ,)
estimated by pointwise fairing.
11/6/2000
Deepak Bandyopadhyay / 258 / 3D Photography
27
Reflectance from Photographs
(Yu, Debevec et al, 1999)
• Estimating reflectance for entire scenes
– Too general a problem, parameterize thus:
• Assume surface can be divided into patches
• Diffuse reflectance function (albedo), varies across a patch
• Specular reflectance function taken as const across a region
• Assume known lighting, calib, geometry known
• Approach - Inverse Global Illumination
– Estimate BRDF for direct illumination - f(u,v,,)
11/6/2000
Deepak Bandyopadhyay / 258 / 3D Photography
28
Reflectance from Photographs
(Yu, Debevec et al, 1999)
• Inverse Global Illumination
– Known Li (measure), Ii (calc fm known light sources) at
every pixel
– Estimate BRDF for direct illumination - f(u,v,i,i,r,r)
• Write BRDF as a constant diffuse term and a specular term
which is a function of incoming & outgoing  and roughness.
• Solve for the constants
(d, s,)
• For indirect illumination - estimate the parameters (and indirect
illumination coeffs with other patches) iteratively
11/6/2000
Deepak Bandyopadhyay / 258 / 3D Photography
29
Case study - Façade
Debevec, Taylor & Malik, 1996
• Modeling architectural scenes from photographs
• Not fully automatic (user inputs blocky 3D model)
– Using blocks leads to fewer params in architectural models
• User marks corresponding features on photo
• Computer solves for block size, scale, camera rotation
by minimizing error of corresponding features
• Reprojects textures from the photographs onto the
reconstructed model
11/6/2000
Deepak Bandyopadhyay / 258 / 3D Photography
30
Arches and
Surfaces of Revolution
Taj Mahal
modeled from
one photograph
11/6/2000
Deepak Bandyopadhyay / 258 / 3D Photography
32
Case study - Digital
Michelangelo Project
• 3D scanning of large statues (SIGGRAPH 00)
• Separate geometry and color scans
– custom rig : laser scanner & camera mounted concurrently
• Range scan post-processing
– Combine range scans from different positions
• Use volumetric modeling methods (Curless, Levoy 1996)
– Fill holes using space carving
11/6/2000
Deepak Bandyopadhyay / 258 / 3D Photography
33
Case study - Digital
Michelangelo Project
• Color scan processing
– Compensate for ambient lighting
• subtract image with & without spotlight
– Subtract out shadows & specularities
– find surface orientation (inverse lighting computation)
– convert color to RGB reflectance (acquire light field)
• Using estimated BRDF of marble
• modeling subsurface scattering
11/6/2000
Deepak Bandyopadhyay / 258 / 3D Photography
34
Digital Michelangelo
Scanning a large object
• calibrated motions
– pitch (yellow)
– pan (blue)
– horizontal translation (orange)
11/6/2000
• uncalibrated motions
– vertical translation
– remounting the scan head
– moving the entire gantry
Deepak Bandyopadhyay / 258 / 3D Photography
35
References
•
•
•
•
•
•
•
[Bouguet98] Bouguet, J.-Y., P. Perona. 3D Photography on your Desk. In
Proc. ICCV 1998
[Bouguet00] Bouguet, J.-Y. Presentation on Desktop 3D Photography, in
SIGGRAPH course notes on 3D Photography, 2000
[Criminisi99] Criminisi, A., I. Reid and A. Zisserman. Single View Metrology.
In Proc. ICCV, pp 434-442, September 1999
[Curless96] Curless, B. and M. Levoy. A Volumetric Method for Building
Complex Models from Range Images. In Proc. SIGGRAPH 1996
[Debevec96] Debevec, P., C. Taylor and J. Malik. Façade - Modeling and
Rendering Architectural Scenes from Photographs. In Proc. SIGGRAPH 1996
[Debevec00a] Debevec, P. Presentation on the Façade, from SIGGRAPH
course notes on 3D Photography, 1999, 2000.
[Debevec00b] Debevec, P., T. Hawkins, C. Tchou, H.P.Duiker, W. Sarokin and
M. Sagar. Acquiring the Reflectance Field of a Human Face. In Proc.
SIGGRAPH 2000.
11/6/2000
Deepak Bandyopadhyay / 258 / 3D Photography
36
More References
•
•
•
•
•
•
[Horn70] Horn, B.K.P. Shape from Shading : A Method for Obtaining the
Shape of a Smooth Opaque Object from One View. Ph.D. Thesis, Dept of EE,
MIT, 1970.
[Kutulakos98] Kutulakos, K. N. and S. Seitz. A Theory of Shape by Space
Carving. URCS TR#692, May 1998, appeared in Proc. ICCV 1999.
[Levoy96] Levoy, M. and P. Hanrahan. Light Field Rendering. In Proc.
SIGGRAPH 1996.
[Levoy00a] Levoy, M., Pulli, K., Curless, B. et al. The Digital Michelangelo
Project - 3D Scanning of Large Statues. In Proc. SIGGRAPH 2000.
[Levoy00b] Levoy, M. Presentation on the Digital Michelangelo Project, in
SIGGRAPH course notes on 3D Photography, 2000.
[Seitz97] Seitz & Dyer. Photorealistic Scene Reconstruction by Voxel
Coloring. In Proc. CVPR 1997, pp. 1067-1073.
11/6/2000
Deepak Bandyopadhyay / 258 / 3D Photography
37
Still More References
•
•
•
•
•
[Seitz00] Seitz, S. SIGGRAPH course notes on 3D photography, 1999, 2000.
[Szeliski93] Szeliski, R. Rapid Octree Construction from Image Sequences.
CGVIP : Image Understanding, vol. 58, no. 1, pp 23-32, 1993.
[Wood00] Wood, D., D. I. Azuma, K. Aldinger, B. Curless, T. Duchamp, D.H.
Salesin and W. Stuetzle. Surface Light Fields for 3D Photography. In Proc.
SIGGRAPH 2000.
[Woodham80] Woodham, R. Photometric Stereo for Determining Surface
Orientation from Multiple Images. Journal of Optical Engineering, vol. 19,
no. 1, pp 138-144, 1980.
[Yu99] Yu, Y., P. Debevec, J. Malik and T. Hawkins. Inverse Global
Illumination - Recovering Reflectance Models of Real Scenes from
Photographs.
11/6/2000
Deepak Bandyopadhyay / 258 / 3D Photography
38