Transcript Axial Light Field for Curved Mirrors: Reflect Your Perspective, Widen

Mitsubishi Electric Research Labs (MERL)
Axial Light Field
Taguchi, Agrawal, Ramalingam & Veeraraghavan
Axial Light Field for Curved Mirrors:
Reflect Your Perspective, Widen Your View
Yuichi Taguchi
Amit Agrawal
Srikumar Ramalingam
Ashok Veeraraghavan
Mitsubishi Electric Research Labs (MERL)
Mitsubishi Electric Research Labs (MERL)
Axial Light Field
Taguchi, Agrawal, Ramalingam & Veeraraghavan
Non-Single Viewpoint Image
(Spherical Mirror + Perspective Camera)
Mitsubishi Electric Research Labs (MERL)
Axial Light Field
Taguchi, Agrawal, Ramalingam & Veeraraghavan
Single-Viewpoint Image (Cube Map, FOV 140)
Mitsubishi Electric Research Labs (MERL)
Axial Light Field
Taguchi, Agrawal, Ramalingam & Veeraraghavan
No Approximation, No Knowledge of Scene Geometry
Mitsubishi Electric Research Labs (MERL)
Axial Light Field
Taguchi, Agrawal, Ramalingam & Veeraraghavan
Single/Non-Single Viewpoint
Perspective
Camera
Hyperbolic
Mirror
Perspective Camera
Single Viewpoint
Perspective
Camera
Virtual
Viewpoint
Spherical
Mirror
Locus of
Viewpoint
(Caustic)
Catadioptric System (Mirror + Camera)
Single Viewpoint
Non-Single Viewpoint
Mitsubishi Electric Research Labs (MERL)
Axial Light Field
Taguchi, Agrawal, Ramalingam & Veeraraghavan
Single-Viewpoint Catadioptric Systems
[Baker & Nayar 99]
Perspective Camera
Orthographic Camera
Foci
Virtual
Viewpoint
Hyperbola/Ellipse
Parabola
• Only a few single-viewpoint configurations
• Other common configurations lead to non-single viewpoint
– Spherical mirror
– Parabolic mirror with perspective camera
– Hyperbolic/elliptic mirror with perspective camera not on foci
Mitsubishi Electric Research Labs (MERL)
Axial Light Field
Taguchi, Agrawal, Ramalingam & Veeraraghavan
Generating Single-Viewpoint Image from
Non-Single Viewpoint Image
• Distortion Correction Approaches
– Use one image
– Single-viewpoint approximation
– Use scene prior [Swaminathan et al. 03]
Scene Prior
(e.g. Plane)
– Generating exact perspective views is impossible without
knowing scene geometry
• Our Approach
– Use multiple images
– Capture all the rays required to generate an exact singleviewpoint image
– Does not require scene prior and 3D reconstruction
Mitsubishi Electric Research Labs (MERL)
Axial Light Field
Taguchi, Agrawal, Ramalingam & Veeraraghavan
Light Field
[Levoy & Hanrahan 96, Gortler et al. 96]
Light Field Plane
Mirror Surface
Virtual Viewpoint
• If we capture all the rays that pass through the virtual
viewpoint, we can generate a single-viewpoint image
– What is the best possible sampling?
Mitsubishi Electric Research Labs (MERL)
Axial Light Field
Taguchi, Agrawal, Ramalingam & Veeraraghavan
Input Images (Axial Light Field)
Mitsubishi Electric Research Labs (MERL)
Axial Light Field
Taguchi, Agrawal, Ramalingam & Veeraraghavan
Input
Output
Copy Circles
Mitsubishi Electric Research Labs (MERL)
Axial Light Field
Taguchi, Agrawal, Ramalingam & Veeraraghavan
Single-Viewpoint Image (Cube Map, FOV 140)
Mitsubishi Electric Research Labs (MERL)
Axial Light Field
Taguchi, Agrawal, Ramalingam & Veeraraghavan
Key Idea
• Rotationally symmetric mirrors
• Capture axial light field
– Move the camera along the mirror axis
– Exact single-viewpoint image generation
without scene prior and 3D reconstruction
– Better sampling than typical planar light field
Symmetry
Axis
Axial
Light Field
Planar
Light Field
Mitsubishi Electric Research Labs (MERL)
Axial Light Field
Taguchi, Agrawal, Ramalingam & Veeraraghavan
Geometric Interpretation
A cone of rays in
virtual camera
(Angle )

d
Rotationally
Symmetric
Mirror

Virtual
Viewpoint
Axial camera [Ramalingam et al. 06]
(All the rays pass through the axis)
A cone of rays
in real camera
(Distance d, Angle )
Mitsubishi Electric Research Labs (MERL)
Axial Light Field
Taguchi, Agrawal, Ramalingam & Veeraraghavan
Light Fields for Symmetric Mirrors in 3D
x-u-v Slice
y
v
u
x
v
u
x
Mitsubishi Electric Research Labs (MERL)
Axial Light Field
Taguchi, Agrawal, Ramalingam & Veeraraghavan
Light Fields for Symmetric Mirrors in 3D
x-u-v Slice
y
y=0
x
v
Virtual
Viewpoint
u
x
Spherical
Mirror
Planar Light Field Sampling
Mitsubishi Electric Research Labs (MERL)
Axial Light Field
Taguchi, Agrawal, Ramalingam & Veeraraghavan
Light Fields for Symmetric Mirrors in 3D
x-u-v Slice
y
y0
x
v
Virtual
Viewpoint
u
x
Spherical
Mirror
Planar Light Field Sampling
Mitsubishi Electric Research Labs (MERL)
Axial Light Field
Taguchi, Agrawal, Ramalingam & Veeraraghavan
Light Fields for Symmetric Mirrors in 3D
x-u-v Slice (x,y)=(uz,vz)
y
x
v
Virtual
Viewpoint
u
x
Spherical
Mirror
Axial Light Field Sampling
Mitsubishi Electric Research Labs (MERL)
Axial Light Field
Taguchi, Agrawal, Ramalingam & Veeraraghavan
Light Fields for Symmetric Mirrors in 3D
u-v Slice (x,y)=(uz,vz)
y
Image
Plane
x
v
Virtual
Viewpoint
u
Spherical
Mirror
Axial Light Field Sampling
Mitsubishi Electric Research Labs (MERL)
Axial Light Field
Taguchi, Agrawal, Ramalingam & Veeraraghavan
Planar LF vs. Axial LF
• Planar Light Field
– Sample a 4D subset of rays
– The camera captures a cone of rays
only when placed on the mirror axis
• Other cameras capture only a few rays
• Axial Light Field
– Sample a 3D subset of rays
– Every camera captures a cone of rays
• Light rays required to generate a
single-viewpoint image are
concentrated in this 3D subset
Mitsubishi Electric Research Labs (MERL)
Axial Light Field
Taguchi, Agrawal, Ramalingam & Veeraraghavan
Axial LF Sampling Parameters
Real
Camera

A cone of rays in
virtual camera
(Angle )
d

Virtual
Camera
A cone of rays
in real camera
(Distance d, Angle )
Mitsubishi Electric Research Labs (MERL)
Axial Light Field
Taguchi, Agrawal, Ramalingam & Veeraraghavan
Axial LF Sampling Parameters
Location of real camera (d)
Ray angle in real camera () []
Resolution
Reduction
Real Camera
FOV
Ray angle in virtual camera () []
Mirror Shapes
Virtual
camera
Ray angle in virtual camera () []
Sphere
Parabola
Concave Sphere
Cone
Mitsubishi Electric Research Labs (MERL)
Axial Light Field
Taguchi, Agrawal, Ramalingam & Veeraraghavan
Single-Viewpoint Image Generation
Copy
Resized
Circles
A cone of rays in
virtual camera
(Angle )
A cone of rays
in real camera
(Angle , Distance d)
Virtual
Perspective
Image
Input
Images
Mitsubishi Electric Research Labs (MERL)
Axial Light Field
Taguchi, Agrawal, Ramalingam & Veeraraghavan
Simulation Results for Different Mirror Shapes
Sphere
Parabola
Concave Sphere
Cone
Input
Far
Input
Near
FOV 24
Output
FOV 140
Limited
Input FOV
Invalid
Z position
Axial Light Field
Mitsubishi Electric Research Labs (MERL)
Taguchi, Agrawal, Ramalingam & Veeraraghavan
Changing Resolution Property
• Change focal length (zoom) for each capture position
– Change resolution property without changing mirror shapes
To achieve resolution property
similar to a perspective camera
Spherical Mirror
Circle radius in real image [pixels]
Mag. factor of focal length
300
5
Constant
focal length
250
4
200
3
150
100
Variable
focal length
50
0
2
1
0
50
100
150
200
250
300
Circle radius in virtual image [pixels]
0
50
100
150
Location of real camera
Mitsubishi Electric Research Labs (MERL)
Comparison
Axial Light Field
Taguchi, Agrawal, Ramalingam & Veeraraghavan
Setup
Robot
Arm
Mirror Ball
Planar Light Field
Axial Light Field
Same Number (25) of Input Images (FOV 32 x 24)
Mitsubishi Electric Research Labs (MERL)
Axial Light Field
Taguchi, Agrawal, Ramalingam & Veeraraghavan
Comparison
Planar Light Field
Axial Light Field
Aliasing /
Ghosting
Output Cube Maps (FOV 140)
Mitsubishi Electric Research Labs (MERL)
Axial Light Field
Taguchi, Agrawal, Ramalingam & Veeraraghavan
Advantages
• Axial light field based catadioptric imaging
– Exact single-viewpoint image generation in wide angle
• Without any scene prior, 3D reconstruction
– For any rotationally symmetric mirror
– Offer changing resolution property
• Without changing mirror shape
– Virtual viewpoints can be varied on the mirror axis
Mitsubishi Electric Research Labs (MERL)
Axial Light Field
Taguchi, Agrawal, Ramalingam & Veeraraghavan
Limitations
• Static scene
• Precise camera motion along the mirror axis
• Implementation using multiple cameras would be difficult
Mitsubishi Electric Research Labs (MERL)
Axial Light Field
Taguchi, Agrawal, Ramalingam & Veeraraghavan
Summary
• Analysis of light rays for rotationally symmetric mirrors
• Axial light field sampling
• Exact single-viewpoint image generation