Transcript week12-math3D2D_B.ppt

CV: 3D to 2D mathematics
Perspective transformation;
camera calibration;
stereo computation;
and more
MSU CSE 240 Fall 2003 Stockman
Roadmap of topics






Review perspective transformation
Camera calibration
Stereo methods
Structured light methods
Depth-from-focus
Shape-from-shading
MSU CSE 240 Fall 2003 Stockman
Review coordinate systems
Camera
or sensor
C
World or
global W
Camera
or sensor
D
Object or
model M
MSU CSE 240 Fall 2003 Stockman
Convenient notation for points
and transformations
This point P
has 2 real
coordinates in
the image
This point P
has 3 real
world
coordinates
in coordinate
system W
This transformation maps each
point in the real world W to a
point in the image I
MSU CSE 240 Fall 2003 Stockman
Current goal
Develop the theory in terms of
modules (components) so that
concepts are understood and can be
put into practical application
MSU CSE 240 Fall 2003 Stockman
Perspective transformation
Camera origin is center of
projection, not lens
X and Y are
scaled by the ratio
of focal length to
depth Z
MSU CSE 240 Fall 2003 Stockman
In next homework & project





fit camera model to image with jig
jig has known precise 3D coordinates
examine accuracy of camera model
use camera model to do graphics
use two camera models to compute
depth from stereo
MSU CSE 240 Fall 2003 Stockman
Notes on perspective trans.




3D world scaled according to ratio of depth
to focal length
scaling formulas are in terms of real numbers
with the same units
e.g. mm in the 3D world and
mm in the image plane
real image coordinates must be further
scaled to pixel row and column
entire 3D ray images to the same 2D point
MSU CSE 240 Fall 2003 Stockman
Goal: General perspective trans
to be developed (accept for now)
Camera matrix C transforms 3D real world point
into image row and column using 11 parameters
MSU CSE 240 Fall 2003 Stockman
The 11 parameters Cij model


internal camera parameters:
focal length f
ratio of pixel height and width
any shear due to sensor chip alignment
external orientation parameters:
rotation of camera frame relative to world frame
translation of camera frame relative to world
The 11 parameters of this model are NOT independent.
Radial distortion is not linear and is not modeled.
MSU CSE 240 Fall 2003 Stockman
Camera matrix via least squares
Minimize the residuals in the
image plane. Get 2 equations
for each pair ((r, c), (x, y, z))
MSU CSE 240 Fall 2003 Stockman
2 equations for each pair
Known 3D points
Here, (u, v) is the point in the
image where 3D point (x,y,z)
is projected. The 11
unknowns d jk form the
camera matrix.
Camera
parameters
MSU CSE 240 Fall 2003 Stockman
Known
image
points
2n linear equations from n pairs
((u,v) (x,y,z))
Standard linear algebra problem; easily solved in
Matlab or by using a linear algebra package.
Often, package replaces b’s with the residuals.
MSU CSE 240 Fall 2003 Stockman
Use a jig for calibration



Get pairings ((r, c) (x, y, z))
Jig has known set of
points
Measure points in
world system W or
use the jig to define
W
Take image with
camera and
determine 2D points
MSU CSE 240 Fall 2003 Stockman
Example calibration data
#
# IMAGE: g1view1.ras
#
#
INPUT DATA
|
OUTPUT DATA
#
|
Point Image 2-D (U,V)
3-D Coordinates (X,Y,Z) |
2-D Fit Data
|
A
95.00 336.00
0.00
0.00
0.00
| 94.53 337.89
B
0.00
6.00
0.00
|
C
11.00
6.00
0.00
|
D
592.00 368.00
11.00
0.00
0.00
| 592.21 368.36
N
O
P
#
501.00 363.00
467.00 279.00
224.00 266.00
9.00
8.25
2.75
0.00
0.00
0.00
0.00
-1.81
-1.81
Residuals X Y
|
|
|
|
-0.21 -0.36
| 501.16 362.78 |
| 468.35 281.09 |
| 224.06 266.43 |
-0.16 0.22
-1.35 -2.09
-0.06 -0.43
CALIBRATION MATRIX
44.84
29.80
-5.504
94.53
2.518
42.24
40.79
337.9
-0.0006832
0.06489
-0.01027
1.000
MSU CSE 240 Fall 2003 Stockman
0.47 -1.89
3D points on jig
Dimensions in inches
MSU CSE 240 Fall 2003 Stockman
Jig set in workspace
Mapping is
established
between 3D
points (x, y, z)
and image
points (u, v)
MSU CSE 240 Fall 2003 Stockman
Other jigs used at MSU


frame with wires and beads placed in
car instead of the driver seat (to do
stereo measurements of car driver)
frame with wires and beads as big as a
harp to calibrate space for people
walking (up to 6 cameras, persons wear
tight body suit with reflecting disks,
cameras compute 3D motion trajectory)
MSU CSE 240 Fall 2003 Stockman
Least squares set up
A
2n x 11
X
=
11
x1
=
MSU CSE 240 Fall 2003 Stockman
B
2n
x1
Least squares abstraction
MSU CSE 240 Fall 2003 Stockman
Justify the form of camera matrix





Another sequence of slides
Rotation, scaling, shear in 3D real world
as a 3x3 (or 4x4) matrix
Projection to real 2D image as 4x4
matrix
Scaling real image coordinates to [r, c]
coordinates as 4x4 matrix
Combine them all into one 4x4 matrix
MSU CSE 240 Fall 2003 Stockman
Other mathematical models
Two camera stereo
MSU CSE 240 Fall 2003 Stockman
Baseline stereo: carefully aligned
cameras
MSU CSE 240 Fall 2003 Stockman
Computing (x, y, z) in 3D from
corresponding 2D image points
MSU CSE 240 Fall 2003 Stockman
2 calibrated cameras view the
same 3D point at (r1,c1)(r2,c2)
MSU CSE 240 Fall 2003 Stockman
Compute closest approach of the
two rays: use center point V
Shortest
line
segment
between
rays
MSU CSE 240 Fall 2003 Stockman
Connector is perpendicular to
both imaging rays
MSU CSE 240 Fall 2003 Stockman
Solve for the endpoints of the
connector
Scaler mult. Fix book
MSU CSE 240 Fall 2003 Stockman
Correspondence problem: more
difficult aspect
MSU CSE 240 Fall 2003 Stockman
Correspondence problem is
difficult




Can use interest points and cross
correlation
Can limit search to epipolar line
Can use symbolic matching (Ch 11) to
determine corresponding points (called
structural stereopsis)
apparently humans don’t need it
MSU CSE 240 Fall 2003 Stockman
Epipolar constraint
With aligned cameras, search for corresponding point is
1D along corresponding row of other camera.
MSU CSE 240 Fall 2003 Stockman
Epipolar constraint for non
baseline stereo computation
Need to know relative
orientation of cameras
C1 and C2
If cameras are not aligned, a 1D search can still be determined for
the corresponding point. P1, C1, C2 determine a plane that cuts
image I2 in a line: P2 will be on that line.
MSU CSE 240 Fall 2003 Stockman
Measuring driver body position
4 cameras were used to measure driver position and
posture while driving: 2mm accuracy achieved
MSU CSE 240 Fall 2003 Stockman