Transcript A Personal Surround Environment: Projective Display with Correction for Display Surface Geometry and Extreme Lens Distortion Tyler Johnson, Florian Gyarfas, Rick Skarbez, Herman Towles and.

A Personal Surround
Environment:
Projective Display with Correction for
Display Surface Geometry and Extreme Lens Distortion
Tyler Johnson, Florian Gyarfas, Rick Skarbez,
Herman Towles and Henry Fuchs
Department of Computer Science
University of North Carolina at Chapel Hill
IEEE VR– March 14, 2007
Multi-Projector Display
2
A Personal Surround Environment
Wide-FOV Projectors
Conventional vs. fisheye lens projector
Conventional Projector
3
Fisheye-Lens Projector
A Personal Surround Environment
Outline
Distortion due to Non-Planar Surfaces
Lens Distortion Correction for Projectors
Results
Contributions and Future Work
4
A Personal Surround Environment
Non-planar Display Surfaces
P
5
A Personal Surround Environment
Geometry Compensation
via Ray Tracing
Display Surface
Projector Image
Projector
Viewer
6
A Personal Surround Environment
Classic Remapping Algorithm
2-Pass Rendering (Raskar ’98)
Pass 1:
Pass 2:
Render application
scene
Remap Pass-1 Image
Projector
Viewer
7
A Personal Surround Environment
2-Pass Rendering
P
Display
Surface
Assumes
no
Geometry
lens
Pinhole
Pinhole
distortion!
Projection
Projection
Pass-1
8
Pass-2
A Personal Surround Environment
Distortion with Wide-FOV Lenses
Input Image
9
Projected Result
A Personal Surround Environment
Previous Applications of
Wide-FOV Projection
Elumens VisionStation
Hemispherical display
screen with fisheye-lens
projector for immersive
display.
Static projector/display
surface relationship.
Distortion correction based
on optical/mechanical
design specifications.
10
A Personal Surround Environment
Previous Applications of
Wide-FOV Projection
Konieczny ’05, “A
Handheld Flexible
Display System”
Allows exploration of volume
data using a deformable
hand-held screen.
Dynamic display surface.
11
A Personal Surround Environment
Our Goals
Our goals were to provide
camera-based calibration for wide-FOV projectors
geometric correction for arbitrary display surfaces
support for tracked viewers
12
A Personal Surround Environment
3-Pass Rendering
Display
Surface
Geometry
Pinhole
Projection
Pass-1
13
Lens Distortion
Correction
Pinhole
Projection
Pass-2
A Personal Surround Environment
3-Pass Rendering Issues
for Wide-FOV Lenses
Fisheye lenses are highly non-linear
Pinhole approximation in Pass-2 leads to
sampling issues during lens distortion
correction in Pass-3.
14
A Personal Surround Environment
Sampling Artifacts
Display Surface
P
3-Pass Rendering
All Passes at Projector
Resolution (1024 x 768)
15
A Personal Surround Environment
Pass-3 Sampling Issues
Fisheye and pinhole lenses have very
different spatial sampling characteristics
Leads to sampling issues during lens distortion
correction.
Over-sampled!
Under-sampled!
Over-sampled!
Pinhole lens
(input)
Fisheye lens
(output)
16
A Personal Surround Environment
Sampling Artifact Reduction
Display Surface
P
Pass 1-2: 4x Projector
Resolution (4096x3072)
Pass 3: (1024x768)
17
A Personal Surround Environment
A New Approach
Remove the pinhole approximation of
the wide-FOV lens.
Use the actual lens model of the projector.
Allows simultaneous compensation for
lens distortion and non-planar surface
geometry.
18
A Personal Surround Environment
Changes to Correction Pipeline
Projector Lens
Model
Display
Surface
Geometry
Pinhole
Projection
Lens Distortion
Correction
Pinhole
Projection
Use actual
projector lens
model to avoid
resampling
artifacts!
Pass-1
19
Pass-2
Pass-3
A Personal Surround Environment
A New Approach
Pre-compute
where projector
rays intersect the
display surface!
Display Surface
Projector Image
Projector
Viewer
20
A Personal Surround Environment
Implementation
Pre-compute the projector to display
surface (ray) mapping
Store this projector-resolution, 2D to 3D map on
GPU as a floating-point texture.
Static map - independent of viewer location.
New Pixel Shader
At each projector pixel:
• Look up <x,y,z> vertex on the display surface.
• Project the vertex into ideal image to obtain output color.
21
A Personal Surround Environment
Results
Display Surface
P
All Passes at Projector
Resolution (1024x768)
22
A Personal Surround Environment
Calibration
Capture structured light
patterns with camera-pair
Calibrate projector
• Linear calibration followed by nonlinear optimization incorporating
projector lens model
Reconstruct surface geometry
• RANSAC-based plane fitting
technique [Quirk, EDT 06]
• General reconstruction techniques
can also be used
23
A Personal Surround Environment
Viewer Tracking
Could apply simple 2D-2D mapping
Applies only to stationary viewer!
Our approach is independent of the viewer
location
Viewer location can be updated each frame
2D - Mapping
Only applicable for
Stationary Viewer
Pass-1
24
Combined Pass-2 & -3 Remapping
A Personal Surround Environment
Heterogeneous 2-Projector Display
Also applicable to
conventional projectors
Display Surface
Fisheye-lens
Projector
Conventional
Projector
25
Hybrid Display with Conventional
and Fisheye-lens Projectors
A Personal Surround Environment
Results
Video
26
A Personal Surround Environment
Contributions
Single-pass geometric remapping algorithm
Correcting for both extreme lens distortion and display surface
geometry
Minimizes sampling and FOV issues with fisheye lenses
Simple GPU shader program implementation
Static 2D:3D mapping stored as 3D floating point texture
Demonstrated applicability
Personal surround environment with a single fisheye lens projector
Conventional or Wide-FOV fisheye AND Heterogeneous displays
Applicable to real-time VR applications with user-head
tracking
27
A Personal Surround Environment
Future Work
Methods to further reduce sampling artifacts
Mip-mapped Pass-1 textures for geometry optimized
sampling.
Application of more general lens models
[Kannala, IEEE TPAMI, Aug. 06]
Real-time, continuous calibration
Accommodate projector or display surface changes while the
application running.
Dynamic vs. static Pass-2 re-map function.
28
A Personal Surround Environment
Thank You
Funding support: ONR N00014-03-1-0589
DARWARS Training Superiority program
VIRTE – Virtual Technologies and Environments program
Elumenati LLC for loan of fisheye-lens projector
29
A Personal Surround Environment