Motion Deblurring Using Hybrid Imaging Moshe Ben-Ezra and Shree K. Nayar Columbia University IEEE CVPR Conference June 2003, Madison, USA.
Download ReportTranscript Motion Deblurring Using Hybrid Imaging Moshe Ben-Ezra and Shree K. Nayar Columbia University IEEE CVPR Conference June 2003, Madison, USA.
Motion Deblurring Using Hybrid Imaging Moshe Ben-Ezra and Shree K. Nayar Columbia University IEEE CVPR Conference June 2003, Madison, USA Image Recording Requires Time Niépce 1827 8 hours exposure Daguerre 1829 1/2 hour exposure Motion Blur is Everywhere Object Motion Camera Motion Stabilized Lenses 1/250 second (< -1 stop) 1/15 second (< -5 stops) Stabilization drifts with time Rotation only Canon Stabilized lens 400mm Blind Image Deconvolution Accurate Point Spread Function (PSF) Needed. Energy ~ 1/ speed Motion Point Spread Function (PSF) H Motion PSF is a Function of: 1. Motion path 2. Motion speed Y X Spatial spread PSF Detector? PSF Detector Camera Can the PSF detector be a small and simple imaging device ? Fundamental Limits of Imaging Photon flux Detector Electron wells Pixel’s Signal Noise Detector’s noise level Temporal resolution (fps) Fundamental Resolution Tradeoff Hybrid imaging system 3 Hi-resolution camera 30 Conventional video camera 130 Low-resolution camera 3M 2048x1536 330K 75K Spatial resolution (pixels) 720x480 320x240 A Hybrid camera enjoys both worlds Overview of Approach Low-Res. camera y Motion Analysis x Same time period PSF Estimation Hi-Res. camera Deconvolution Global Motion From Low Resolution Detector I I I arg min u v y t ( u ,v ) x 2 Objective function (Optical flow constraint) Rotation Translation u cos sin xx v sin cos y y 0 1 1 0 1 Lucas Kanade Simulations: Motion Accuracy from Low- Res. Images Average Motion Error in Pixels =3 =9 = 27 = 81 640x640 (1:1) 0.01 0.01 0.02 0.04 320x320 (1:4) 0.03 0.04 0.05 0.1 160x160 (1:16) 0.03 0.04 0.07 0.4 80x80 (1:64) 0.13 0.21 0.39 2.6 Noise Resolution Constraints on Continuous PSF Energy conservation constraint: h( x, y)dx dy 1 xy Constant flux t t assumption: t Smoothness constraint: h(x(t), y(t))dt t tend t start Path is continuous and twice differentiable PSF Estimation from Computed Motion y f1 f2 f3 y f6 f4 f5 f1 f2 f3 f4 f5 … Frame 2 f6 Frame 5 x h4 h2 h h3 x h4 h5 h2 h y Frame 2 … Frame 5 h3 h5 y Frame 2 … Frame 5 x Deconvolution of High Resolution Image Standard iterative ratio-based algorithm* Image estimate PSF Error (0) ˆ O ( x) I ( x) Oˆ ( k 1) I ( x ) ˆ ( x) O ( x) S ( x) , (k ) ˆ S O (k ) Guaranties non-negative pixel result * Richardson [72] Lucy [74] Designs for Hybrid Imaging A rig of two cameras Using a beam splitter Using a special chip Our Prototype: Rig of Two Cameras Primary detector (2048x1536) Secondary detector (360x240) Resolution ratio of 1 : 36 Example 1 - Blurred Hi-Res Image f = 633mm, Exp. Time 1 Sec (> -9 stops) PSF Estimation from Motion Estimated PSF Low resolution sequence. 0.06 Y (Pixels) 90 10 10 X (Pixels) f = 633mm, Exp. Time 1 Sec 130 0.001 Deblurred Image f = 633mm, Exp. Time 1 Sec Example 1 - Comparison Tripod image (Ground Truth) Blurred image Deblurred image f = 633mm, Exp. Time 1 Sec Example 2 - Blurred Night Image f = 884mm, Exp. Time 4 Sec (> -11 stops) PSF Estimation from Motion Low resolution sequence. 0.003 Y (Pixels) 30 10 0.001 10 X (Pixels) f = 884mm, Exp. Time 4 Sec 60 Deblurred Night Image f = 884mm, Exp. Time 4 Sec Example 3 - Comparison Tripod image (Ground Truth) Blurred image Deblurred image f = 884mm, Exp. Time 4 Sec Object Deblurring Problem Moving objects blend into the background Hybrid Imaging Solution (simulated) Requires clear high-resolution background image Quantifying The Affect of Motion Blur Empirical tests: RMS error. Volume of Solutions (Linear Model): Input Images High-Resolution Image 1 1 y Ax z x A y A z Blur Uncertainty Decimation (Quantization) Volume of Solutions 1/det(A)