Transcript Computational Plenoptic Imaging - University of British Columbia

Computational Plenoptic Imaging
Gordon Wetzstein1 Ivo Ihrke2 Douglas Lanman3 Wolfgang Heidrich1
1University
of British Columbia
2Saarland
University
3MIT
Media Lab
VI. Multiplexing Time
Eurographics 2011 – State of the Art Report
EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging
1
History – Eadweard Muybridge 1830-1904
http://en.wikipedia.org/wiki/Eadweard_Muybridge
EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging
2
History – Étienne-Jules Marey 1830-1904
www.wikipedia.org
EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging
3
VI.I Time Lapse Photography
EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging
4
BBC Time Lapse – Look It!
• Long exposures to avoid temporal aliasing
ND 3.0 filter, f22, 1 minute exposure
EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging
5
VI.II High-Speed Imaging
EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging
6
High-Speed Cameras
Vision Research Phantom Flex (CMOS)
2570 fps at HD resolution
Shimadzu HyperVision HPV-2 (CCD)
one million fps at 312x260 pixels
Photron FASTCAM SA5 (CMOS)
7500 fps at megapixel resolution
one million fps at 64x64 pixels
Casio Exilim Series (consumer cam)
1000 fps at reduced resolution
EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging
7
Assorted Pixels
[Narasimhan & Nayar 05]
EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging
8
Temporal Mosaic with DMD
DMD aligned with CCD in microscope
[Bub et al. 10]
EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging
9
Non-Destructive Sensor Readout & Pixim
www.pixim.com
Cypress Semiconductor LUPA 3000
3 megapixels, 485 fps
EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging
10
Coded Rolling Shutter
EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging
[Gu et al. 10]
11
Reinterpretable Imager
• Moving pinhole over time in aperture
• Capture with light field camera
EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging
[Agrawal et al. 10]
12
Bullet Time Effect
from ‘The Matrix’
EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging
13
Stanford Multi-Camera Array
EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging
[Wilburn et al. 04]
14
Coded Temporal Sampling
EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging
[Agrawal et al. 10]
15
High-Speed Illumination – Electronic Strobes
Harold ‘Doc’ Edgerton
1903-1990
EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging
16
Temporal Dithering with DLP Illumination
EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging
[Narasimhan et al. 08] 17
Coded Strobing Photography
EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging
[Reddy et al. 11]
18
Streak Cameras
C5680
$200K
www.hamamatsu.com
EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging
19
VI.I Motion Deblurring
EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging
20
Motion Deblurring Overview
• Motion blur is velocity-dependent
• Can be described as convolution, where
– Kernel shape is motion trajectory
– Trajectory is modulated by exposure function
http://en.wikipedia.org/wiki/Motion_blur
EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging
21
Deconvolution is Still Hard
• Again – problems:
– Camera noise
– Spatially varying kernel (velocity-dependent)
– Unknown motion trajectory
– Ill-posed problem, kernel of box integration
function is not invertible (optical cancellation
of image frequencies)
EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging
22
Approaches to Improve Motion Deblurring
• Make PSF invertible  coded exposure
• Make PSF velocity-invariant  shiftinvariant deconvolution
• Automatize PSF estimation
EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging
23
Flutter Shutter
[Raskar et al. 06]
EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging
24
Optimal Motion PSFs
• Optimality criteria PSF invertibility &
estimation
[Agrawal & Xu 07]
EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging
25
Motion Invariant Photography
• Engineer PSF to be motion invariant
• Only for 1D motion
EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging
[Levin 08]
26
Hybrid Cameras
• Combined high-speed low-quality & lowspeed high-quality camera
[Ben-Ezra & Nayar 04]
Input images
Deblurred Result
EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging
Computed PSF
Ground Truth
27
Motion Blur in Video
• Coded exposure &
super-resolution in
successive video frames
[Agrawal et al. 09]
EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging
28
Next: Further Light Properties
EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging
29
Flexible Voxels
• Flexible space-time resolution as postprocessing
EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging
[Gupta et al. 10]
30
Synthetic Shutter Speed Imaging
[Telleen 07]
• Combine multiple short exposures to
reduce noise
• Align with optical flow
EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging
31
Hybrid Cameras
[Tai et al. 08]
• Motion deblurring & super-resolution
EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging
32
Hybrid Cameras
• Motion deblurring & depth from two lowresolution high-speed camers
[Li et al. 08]
Input images
Deblurred result
EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging
Recovered Depth
33
Analysis
• Analysis of optimal coded, single image
deblurring
• MIP becomes worse when velocities
exceed expectations
Coded Exposure
Motion Invariant Photography
[Agrawal & Raskar 09]
EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging
34