Transcript Image Enhancement Method via Blur and Noisy Image Fusion Marius Tico, Kari Pulli Nokia Research Center Palo Alto, CA, USA.

Image Enhancement Method via
Blur and Noisy
Image Fusion
Marius Tico, Kari Pulli
Nokia Research Center
Palo Alto, CA, USA
Outline
• Introduction
• Related work
• Proposed method
• Photometric calibration
• Luminance fusion
• Color fusion
• Experiments and examples
• Conclusions
2
Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli
Introduction
• Image capturing in dim light is challenging, especially for miniature cameras
• Tuning camera parameters tradeoffs between different quality factors
• Aperture increase: ensures more light but reduces depth of field
• ISO sensitivity increase: amplifies the noise
• Exposure time increase: may result in motion blur
3
Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli
Related work
• De-noise a single image captured with short exposure (and high ISO):
• e.g., [Donoho and Johnstone, 1994], [Starck, Candes, and Donoho, 2002], [Portilla, et
al. 2003]
• Due to short exposure and quantization color may not be recovered
• De-blurring a single image captured with long exposure time (and small ISO)
• Blind image de-convolution [You and Kaveh '96 & '99], [Chan and Wong '98], [Fergus
’06], [Shan ‘08]
• Computationally complex, plus reliability problems
• Assume spatially invariant blur PSF
• Fusing blurry / noisy image pairs [Tico, et al. ‘06], [Yuan. et al. ‘07]
• Avoid blind de-convolution by estimating the blur PSF from the two input images
• Assume blur PSF is spatially invariant
• Using additional hardware
• Extra video camera for motion estimation [Ben-Ezra, Nayar ‘04]
• Flash: effective only for close objects, and change the mood of the scene
• Opto-mechanical stabilizer systems: inefficient for long exposure times
4
Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli
The proposed method
• Short exposed frame: noisy but not affected by motion blur
• Long exposed frame: better colors, less noise, but blurry
Time
Short exposed: dark, noisy and less color
5
Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli
Long exposed: good color but blurry
The block diagram of the proposed solution
6
Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli
Image fusion
Registration
Photocalibration
Photometric calibration
• Build the joint histogram (comparagram)
7
Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli
Photometric calibration (cont’d)
• Identify most likely correspondences (xi,yi) between pixel values in the two images
8
Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli
Photometric calibration (cont’d)
• Estimate the Brightness Transfer Function (BTF)
9
Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli
Photometric calibration (cont’d)
Calibration curves
Short exposed
Long exposed
10
Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli
Calibrated short exposed
Image fusion
11
Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli
Image fusion
Registration
Photocalibration
Luminance fusion
• Formally we can write the following model for the two images
• By applying an orthogonal wavelet transform the model becomes
• Taking advantage of the de-correlation in the wavelet domain we propose a
MMSE diagonal estimator of the form
where
• By minimizing the mean square error
results the expression for the optimal weight
12
Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli
Luminance fusion (cont’d)
• Prefer the noisy image near edges, and the blurry image in smooth areas
• Edges can be detected based on the difference between the two images
13
Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli
Different levels of blur
24.61dB
14
33.76dB (3x3)
29.81dB (5x5)
35.09dB
34.03dB
Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli
25.58dB (11x11)
33.50dB
22.78dB (21x21)
33.40dB
Different levels of noise
27.80dB (7x7)
15
28.13
24.61dB
35.32dB
33.62dB
Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli
22.12dB
32.47dB
20.21dB
31.48dB
Comparative simulations
24. 92dB
Wavelet HT
24.73dB
Curvelet HT
27.43dB
31.05dB
Wiener
29.44dB
31.19dB
21.02dB (7x7)
16
Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli
[Portilla, et al. 2003]
Comparative simulations (cont’d)
• The CPU times measured on Intel Core 2 Duo 2.20GHz
Proposed method
31.19dB
17
CPU Time: 0.8 sec
Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli
Portilla, et al. 2003
31.05dB
CPU Time: 31 sec
Noise variance estimation
• Noise variance in the photo-calibrated image
= noise variance in the short exposed image
= brightness transfer function
= short exposed image in pixel
Short exposed - calibrated
18
Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli
Noise variance
Color fusion
• Emphasize colors from long exposed image except the areas where the long
exposed image is saturated
• Weighting functions
• Saturation weight function (left)
• Blurriness weight function (right)
19
Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli
Color fusion (cont’d)
• Example of color weighting in accordance to the two rules (saturation and
blurriness )
Short exposed
20
Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli
Color weight
Long exposed
Example
Short exposed
21
Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli
Result
Long exposed
Example (local blur)
Short exposed
22
Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli
Result
Long exposed
Example (local blur)
De-blurring
23
Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli
Result
Long exposed
Low Light Imaging
Short exposed: 1/30
sec, ISO 100
24
Photometric aligned
short exposed
Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli
Long exposed:
1 sec, ISO 200
Output
24
Conclusions
• We proposed a new approach of image restoration by fusing two differently
degraded images
• Short exposed image affected by noise
• Long exposed image that may be affected by motion blur
• In contrast to previous blurred/noisy image fusion our approach is not applying
convolution on the blurry image
• The main advantages:
de-
• Can deal with spatially variant blur due to parallax, or object motion
• Lower computational complexity
• Since the proposed approach is not dependent of blur spatial invariance it can be
used also for fusing images with different aperture
• Small aperture image, affected by noise but capturing a large depth of field
• Large aperture image, less noisy but affected by blur due to narrow depth of field
25
Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli
Thank you!
26
Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli
References
• De-noise a single image captured with short exposure (and high ISO)
• Several de-noising methods available in the literature, e.g.,
• D. L. Donoho and I. M. Johnstone, “Ideal spatial adaptation by wavelet shrinkage,” Biometrika, vol. 81, pp. 425–455, 1994.
• Jean-Luc Starck, Emmanuel J. Candes, and David L. Donoho, “The Curvelet Transform for Image Denoising,” IEEE Trans. on
Image Processing, vol. 11, no. 6, pp. 670–684, 2002.
• J. Portilla, V. Strela, M. Wainwright, E.P. Simoncelli, “Image denoising using scale mixtures of Gaussians in the wavelet
domain”, IEEE Trans. on Image Processing 12, No. 11, 1338–1351, 2003.
• Some are too complex a mobile device computational power
• De-blurring a single image captured with long exposure time (and small ISO)
• Blind image de-convolution
• Q. Shan, J. Jia, and A. Agarwala, “High-quality motion deblurring from a single image”, SIGGRAPH, 2008.
• R. Fergus, B. Singh, A. Hertzmann, S.T. Roweis, and W.T. Freeman, “Removing camera shake from a single photograph”,
SIGGRAPH, 2006.
• High complexity and insufficient robustness for consumer applications
• Using additional camera for motion estimation
• M. Ben-Ezra, S.K. Nayar, "Motion-based motion deblurring", IEEE Trans. on PAMI, 26, No. 6, 689-698, 2004
• Using specially designed CMOS sensors
• X.Liu, A. Gamal, "Synthesis of High Dynamic Range Motion Blur Free Image From Multiple Captures", IEEE Trans. on Circuits
and Systems I: Findamental Theory and applications, vol. 50, no. 4, 530-539, 2003.
• Fusing blurry / noisy image pairs
• Marius Tico, Mejdi Trimeche, and Markku Vehvil¨ainen, “Motion blur identification based on differently exposed images”, ICIP,
2006.
• Lu Yuan, Jian Sun, Long Quan, and Heung-Yeung Shum, “Image deblurring with blurred/noisy image pairs,” SIGGRAPH 2007.
27
Image enhancement via blurred & noise image fusion / Marius Tico, Kari Pulli