Transcript Automatic Compensation for Camera Settings for Images Taken under Different Illuminants

Automatic Compensation for
Camera Settings for Images Taken
under Different Illuminants
Cheng Lu and Mark S. Drew
Simon Fraser University
{clu, mark}@cs.sfu.ca
1
Flash/No-flash Imagery –
What About Camera Settings?
(or, more generally, pairs of images
with two different illuminants).
Growing body of research on combining flash/no-flash
image pairs to carry out tasks in:
- Computer Vision and in
- Color Science
2
One use: “Removing Shadows using Flash/Noflash Image Edges”
[Lu, Drew, & Finlayson, ICME 2006]
1. Ambient Shadows in Images
Problem: How to remove shadows from
images?
 One answer: 2 images
, one under
ambient lighting, & another under flash.

+
Under Ambient: Image “A”.
Under Both: Image “B”.
3
2. The Key: Pure-Flash Image
2. The Key: Pure-Flash Image

(
The light from “A” is also in “B”.
 The ambient light from “A” is also in “B”.
Therefore if we subtract the two, we have
Therefore
if we subtract
“F”:
the pure-flash
image. the two, we have
“F”: the pure-flash image.
+
+
)
Under Ambient: Image “A”.
Under Flash: Image “F”.
-
-
=
=
Under Both: Image “B”.
But problem: different shadows.
4
3. Problems with Pure-Flash Image


Flash image “F” has extra shadows.
Also, other well-known problems:
 Harsh or unpleasing effects; pixel saturation; strong
interreflections.
 Rapid illumination fall-off with distance: “tunnel effect”.
So can we use flash-image “F” information in the
ambient-shadows regions of “A” to see into the
ambient shadows?
5
(
4. Strategy

Simplified model for lighting change.


Go to log domain: lighting effects are additive.
Gives a chromaticity 2-space with axes:
Promoted by “spectral sharpening” transform of colors.
Along illuminant-change direction
Along gray axis
Provides excellent separation of
shadow/nonshadow regions.
 Regress pixel data from “F” to “A”, to set scale.
 Insert edges from “F” into “A”; invert Poisson
eqn. to fill in shadow.

)
6
But need to ensure that
really gives just the image under pure-flash
lighting.
If settings are different, won’t work,
without compensation!
7
Strategy:
 Wish to compensate for
exposure time,
ISO,
aperture,
focal length,
white balance.
 Can use a 2nd-order “masking model” (i.e.,
polynomial) on such parameters
 How do we know how to compensate?
 Make shadow disappear for difference of adjusted
images, by matrixing,
 Map pairs of settings to matrix via masking model.
8
Strategy, cont’d:
 Simplify matrix: Adjust magnitude in each color
channel so as to eliminate shadow in:
(with-flash) – (no-flash),
over large set of image pairs.
 Train polynomial model.
 Apply polynomial model to new image pairs.
9
Assumptions:
 Additivity and proportionality of (transformed)
camera parameters
 2nd order polynomial model
9 parameters.
(Compare CMY overprinting:
X0
2
log
 aC  bM  cY  dC
X
2
2
 eY  fM  gCM  hCY  iMY
)
10
Example of image pairs:
Ambient light (“A”)
No scaling
Scaled to max=255
Ambient + flash
(“Both”, “B”)
11
Now subtract:
No, see shadow in
pure-flash image!
So use in-shadow,
out-of-shadow
regions to obtain 3
color-channel
multipliers 
12
 We need 3-vector of scaling
coefficients A A so boxes match, in
difference image.
Call in-shadow region “s”, out-ofshadow “ns”:
13
Now what is M: A A as a function of
camera settings?
use polynomial model (like for printers) -uses log’s and assumes additivity and
proportionality of values.
Parameters:
14
Training:
1. Fix focal length, use tripod.
2. Use “auto” setting; and acquire actual settings
used from stored image meta-data.
3. Use EV (exposure value) = same for all shutter
speed/aperture combinations that give same
exposure. In APEX system (Additive
Photographic Exposure System), EV=AV+TV;
AV=ApertureValue=log2f2, TV=TimeValue=-log2t
4. ISO automatic
5. White balance  encapsulate the effect of
white balancing by using use the mean
value for each RGB channel in the masking
model.
15
6. Ok, we generate values in M: A A by
selecting in/out-of-shadow areas by hand.
What model should we use for mapping
settings to M?  Use log’s of ratios, in 2ndorder model:
9 parameters
a1,a2,a3,b1,b2,b3,
c1,c2 c3,
so use leastsquares.
Then apply same
model to new
image pair.
16
()  (
3N x 1
)( )
3N x 9
9x1
??
17
Experiments
• 5 lighting sources:
– Direct sunlight, cloudy daylight,
a tungsten light lamp and
incandescent lamp, and xenon
flash light.
• Images captured in 5
situations:
125 training image pairs;
125 tests using
take-one-out re-calc. of M:
re-compute 9 param’s,
predict M, apply.
Sophisticated experimental setup
18
:
Ambient
images
Ambient+flash
images
Pure
flash
images
Success! – no shadows
19
Thanks!
To Natural Sciences and
Engineering Research Council of Canada
20