Transcript Radiometric Self Calibration Tomoo Mitsunaga Shree K. Nayar Hashimoto Signal Processing Lab. Sony Corporation Dept.

Radiometric Self Calibration
Tomoo Mitsunaga
Shree K. Nayar
Hashimoto Signal Processing Lab.
Sony Corporation
Dept. of Computer Science
Columbia University
CVPR Conference
Ft. Collins, Colorado
June 1999
Problem Statement
How well does the image represent the real world?
Image M2
(Low exposure)
Image M1
(High exposure)
Usual imaging systems have :
• Limited dynamic range
• Non-linear response ( M 2  M1 )
June/1999
CVPR99
2
Scene Radiance and Image Irradiance
E
L
Radiance
Irradiance
 d
2
Image irradiance :
d
E  L   cos   kL    
4h
2
Ideal camera response :
I  E t
Exposure :
2
4
Aperture area
2
d
e     t
2
I  kL  e
June/1999
CVPR99
3
Scene Radiance and Measured Brightness
Video
Image
Formation
Scene
radiance
L
Image
Exposure
linear
CCD
Scaled
radiance
I
Camera
Electronics
Digitization
I  f (M )
M  f 1 ( I )
Measured
brightness
M
Photo
Image
Formation
Image
Exposure
Film
Film
Development
Scanning
f (M) : The radiometric response function
June/1999
CVPR99
4
Calibration with Reference Objects
The scene must be controlled
• The reflectance of the objects must be known
• The illumination must be controlled
June/1999
CVPR99
5
Calibration without Reference Objects
• Differently exposed images from an arbitrary scene
• Recover the response function from the images
• Calibrate the images with the response function
1
0.8
Input Images
0.6
Response function
0.4
0.2
0
0
0.2
0.4
0.6
0.8
1
High dynamic range radiance image
June/1999
CVPR99
6
Previous Works
• Mann and Picard (95) :
– Take two images with known exposure ratio R
– Restrictive model for f :
M a  bg I
– Find parameters a, b, g by regression
• Debevec and Malik (97) :
– General model for f : only smoothness constraint
– Take several (say, 10) high quality images
– At precisely measured exposures (shutter speed)
June/1999
CVPR99
7
Obtaining Exposure Information
We have only rough estimates
• Mechanical error
• Reading error (ex. F-stop number)
June/1999
CVPR99
8
Radiometric Self-Calibration
Works with roughly estimated exposures
• Inputs :
– Differently exposed images
– Rough estimates of exposure values
– ex. F-stop reading
• Outputs :
– Estimated response function
– Corrected exposure values
June/1999
CVPR99
9
A Flexible Parametric Model
1
High order polynomial model :
0.8
video
N
I  f ( M )   cn M n
0.6
f (M)
n 0
s
0.4
posi
0.2
Parameters to be recovered :
nega
• Coefficients cn
• Order N
0
0
0.2
0.4
0.6
0.8
1
M
f(M) of some
popular imaging products
June/1999
CVPR99
10
Response Function and Exposure Ratio
Images: q = 1,2,….Q , Pixels: p = 1, 2, …..P
Exposure ratio: Rq ,q 1 
eq
eq 1
, Rq ,q 1 
L p k p eq
L p k p eq 1

I p ,q
I p ,q 1
n0 cn M p,q
N
Using polynomial model :

N

f ( M p ,q )
f ( M p ,q 1 )
n
c M p ,q 1
n

Rq ,q 1
n 0 n
Thus, we obtain ...
Q 1 P
Objective function :
June/1999

n
n
    cn M p ,q  Rq ,q 1  cn M p ,q 1 
q 1 p 1  n 0
n 0

N
CVPR99
N
2
11
An Iterative Scheme for Optimization
Rough estimates
Rq,q+1(0)
Rq,q+1(i)
Optimize
for Rq,q+1
Optimize
for f
f (i)
f (i )  f (i 1)   M
Optimized f and Rq,q+1
June/1999
CVPR99
12
Evaluation : Noisy Synthetic Images
1
14
16
0.8
25
34
48
51
64
0.6
72
91
99
f (M)
14
16
0.4
25
34
48
51
64
0.2
72
91
99
0
0
0.2
0.4
0.6
0.8
1
M
Solid : Computed response function
Dots : Actual response function
June/1999
CVPR99
13
Evaluation : Noisy Synthetic Images (cont’d)
10
8
Percentage
Error in
Computed
Response
Function
6
4
2
0
0
10
20
30
40
50
60
70
80
90
100
Trial Number
Maximum Error : 2.7 %
June/1999
CVPR99
14
Computing a High Dynamic Range Image
• Calibrating by the response function f (M )
• Normalizing by corrected exposure values e~
• Averaging with SNR-based weighting w(M )
M p ,1
M p,2
f (M )
f (M )
I p ,1
I p,2
1~
e1
~
I p ,1
1~
e2
~
I p,2
~
w
(
M
)
I
q1 p,q p,q
Q

Q
~
I p ,Q
M p ,Q
June/1999
f (M )
I p ,Q
q 1
Ip
w( M p ,q )
1~
eQ
CVPR99
15
Results : Low Library (video)
Captured images
1
0.8
0.6
I
0.4
0.2
Calibration chart
0
0
0.2
0.4
0.6
0.8
1
M
Computed response function
June/1999
CVPR99
16
Results : Low Library (video)
Captured images
Computed radiance image
June/1999
CVPR99
17
Results : Adobe Room (photograph)
Captured images
1
0.8
0.6
I
0.4
0.2
0
0
0.2
0.4
0.6
0.8
1
M
Computed response function
June/1999
Computed radiance image
CVPR99
18
Results : Taos Clay Oven (photograph)
Captured images
1
0.8
0.6
I
0.4
0.2
0
0
0.2
0.4
0.6
0.8
1
M
Computed response function
June/1999
Computed radiance image
CVPR99
19
Conclusions
A Practical Radiometric Self-calibration Method
• Works with
– Arbitrary still scene
– Rough estimates of exposure
• Recovers
– Response function of the imaging system
– High dynamic range image of the scene
Software and Demo
http://www.cs.columbia.edu/CAVE/
June/1999
CVPR99
20