Applications of Light Polarization in Vision

Lecture #18 Thanks to Yoav Schechner et al, Nayar et al, Larry Wolff, Ikeuchi et al

Separating Reflected and Transmitted Scenes Reconstructing Shape of Transparent Objects Removing Haze and Underwater Scattering Effects Removing Specularities

Separation of Diffuse and Specular Reflections Diffuse surfaces : No (or minimal) Polarization All light depolarized due to many random scattering events inside object.

Specular Surfaces: Strong Polarization (even though partially polarized) Smooth/Rough Surfaces: The degree of polarization decreases with roughness.

Active Illumination

•Completely remove specular reflections using polarized light when the filters are 90 degrees apart.

•Commonly used in industrial settings.

Passive Illumination

•Most illumination from sources (sun, sky, lamps) is unpolarized.

•Merely using a polarizer will not remove specular reflections completely.

I_min is not equal to I_d (diffuse component)

I

max

I

min camera polarizer Polarization Measurements  max Polarization vector determination 3 general measurements suffice  max 180 o

Determining the Polarization Cosine Curve Using Vector Notation: Three measurements suffice to determine the cosine curve.

Degree of Polarization •Varies between 0 and 1.

•If zero, then there is no polarization  Only diffuse component present.

•If one, only specular component present.

•If degree of polarization does not change as polarizer is rotated, then there is no guarantee that specular component is completely removed (I_sc may still be present).

Fresnel Ratio •I_sc and I_sv depend on refractive index and angle of incidence.

•I_sc and I_sv are related to fresnel coefficients: is fresnel coefficient perpendicular to plane of incidence is fresnel coefficient parallel to plane of incidence

Fresnel Ratio Metals Dielectrics •Hard to separate diffuse and specular parts for metals.

•Easier for dielectrics (good for non-normal incidences).

Brewster angle

Dichromatic Model for Removing Specularities Completely •Specularities are only reduced in intensity using polarization.

•They are removed completely only for the Brewster angle of incidence.

•Nayar et al. use additional color constraints in dichromatic model to remove reflections completely.

• Assume a local patch where the highlight and its surrounding area have the same diffuse component.

Semi-Reflections

•Both Reflected and Transmitted light are polarized.

•But they are polarized differently.

•They depend on the orientation of the transparent layer.

•Reflections are removed completely only at Brewster Angle of Incidence.

Transparent Layers

window camera

Semi-Reflections

transmitted scene

I T

reflected scene

I R

  window camera

reflected scene

I

r

transmitted scene

I

t

0.8

0.6

0.4

0.2

20

r

40

r

60 80  window  

r I

r

polarizer

r I

r

camera Optical coding Yoav Schechner, Joseph Shamir, Nahum Kiryati ‘99

reflected scene

I

r

transmitted scene

I

t

0.8

0.6

0.4

0.2

t=1-r

20

r

40

r

60 80  window  

t I

t

r I

r

t I

t

r I

r

polarizer camera Optical coding Yoav Schechner, Joseph Shamir, Nahum Kiryati ‘99

Experiment

transmitted scene

I

t

reflected scene

I

r

Yoav Schechner, Joseph Shamir, Nahum Kiryati ‘99 window observed image

Optical coding

transmitted scene

I

t

reflected scene

I

r

observed image Yoav Schechner, Joseph Shamir, Nahum Kiryati ‘99 window

I I

= [

r I

r

+

t I

t

] / 2

Optical coding

transmitted scene

I

t

reflected scene

I

r

observed image Yoav Schechner, Joseph Shamir, Nahum Kiryati ‘99 window

I I

= [

r I

r

+

t I

t

]/ 2

Digital decoding

2 Linear equations

I

= [

r I

r

+

t I

t

]/ 2 Solve for 2 unknowns:

I R

,

I T I

= [

r I

r

+

t I

t

] / 2 Window at 27 o

I I

Reflected

I R r

2

_ _

2

r r

_

2

r

_ _

2

r r I I

Transmitted

I T r

2

_

r r

_

r

2

_

r r

Yoav Schechner, Joseph Shamir, Nahum Kiryati ‘99

17 o 27 o 37 o Transmitted

I T

Reflected

I R

negative crosstalk 0.4

0

_

0.4

_

0.8

10 20 30 40 50  positive crosstalk The inclination of an invisible surface

clear day

Imaging through Haze

moderate haze very hazy Recover:    Object + haze layers Scene structure Info about the aerosols Previous work Pure image processing Grewe & Brooks ’98, Kopeika ’98 Oakley & Satherley ’98 Physics based Nayar & Narasimhan ’99 Polarization filtering Shurcliff & Ballard ’64

Instant Dehazing:

Yoav Schechner, Srinivasa Narasimhan, Shree Nayar

Imaging through Haze

object radiance

R

camera Airlight

A

direct transmission

T

scattering

Instant Dehazing:

Yoav Schechner, Srinivasa Narasimhan, Shree Nayar

object radiance

R

camera Airlight

A

direct transmission

T

scattering

I

total (

x

,

y

) 

T

(

x

,

y

) 

A

(

x

,

y

) 1 0

e

 

z



R

(

x

,

y

)

z

1 0 1 

e

 

z



A



z

  

z

is a function of (

x,y

)

C o l o r

Multiplicative & additive models - similar dependence

Polarization and Haze

camera polarizer

A A

direct transmission Plane of rays determines airlight components

A

Airlight degree of polarization

p



A A

_

A

+

A

>

A A A

= 0 =

A

polarized unpolarized

p

= 1

p

= 0 Along the line of sight, polarization state is distance invariant Assume: The object is unpolarized

T

/ 2 @ all orientations

Instant Dehazing:

Yoav Schechner, Srinivasa Narasimhan, Shree Nayar

Trivial case

I I

Life is tough…

… still, there is a dominant polarization

Instant Dehazing:

Yoav Schechner, Srinivasa Narasimhan, Shree Nayar

Experiment

Best polarized image

I

=

T

/2 +

A

Instant Dehazing:

Yoav Schechner, Srinivasa Narasimhan, Shree Nayar

Experiment

Worst polarized image

I

=

T

/2 +

A

Instant Dehazing:

Yoav Schechner, Srinivasa Narasimhan, Shree Nayar

Model

object radiance

R

airlight

A

transmission

T

camera 2 input images:

I



T

/ 2 

A I



T

/ 2 

A

transmission airlight

T



R e



βz A



A

 1 -

e



βz

polarization degree

p



A A

_

A

+

A

Recovery

depth

e

 

z

 1  (

I



I A

 ) /

p

radiance

R

 (

I



I

)

e

  (

I βz



I

) /

p

for known

p

,

A



Model

camera 2 input images:

I



T

/ 2 

A I



T

/ 2 

A

transmission airlight

T



R e



βz A



A

 1 -

e



βz

polarization degree

p



A A

_

A

+

A

saturated airlight

A

 airlight polarization

p

Recovery

depth

e

 

z

 1  (

I



I A

 ) /

p

radiance

R

 (

I



I

)

e

  (

I βz



I

) /

p

for known

p

,

A



I

Best polarized image

Dehazing Experiment

Instant Dehazing:

Yoav Schechner, Srinivasa Narasimhan, Shree Nayar

R

Dehazed image

Dehazing Experiment

Instant Dehazing:

Yoav Schechner, Srinivasa Narasimhan, Shree Nayar

Range map depth

e

 

z

 1  (

I



pA



I

) component images

I

(

x,y

) ,

I

(

x,y

) Airlight saturation polarization

A



p

log 

z

(

x, y

)

I

Best polarized image

Dehazing Experiment

Instant Dehazing:

Yoav Schechner, Srinivasa Narasimhan, Shree Nayar

R

Dehazed image

Dehazing Experiment

Instant Dehazing:

Yoav Schechner, Srinivasa Narasimhan, Shree Nayar

Range map

Instant Dehazing:

Yoav Schechner, Srinivasa Narasimhan, Shree Nayar

veiling light signal S B

I

total 1 0

e

 

z



L

effective object

z

+ 1 0 1 

e

 

z



B

  

z

is a function of (

x,y

)

C o l o r

 - object with blur

z

Hypothesis, 4 Decades Old

Lythgoe & Hemmings, 1967 (

Nature

) : “Many invertebrates are able to distinguish the plane of polarized light. Does this enable them to

see further

underwater?” Lythgoe, 1972 (

Handbook Sensory Physiol

) : “…there is a strong possibility that it [polarization] could be useful for improving the visibility of under water.

”

distant objects

, especially

Hypothesis, 4 Decades Old

Lythgoe & Hemmings, 1967 (

Nature

) : “Many invertebrates are able to distinguish the plane of polarized light. Does this enable them to

see further

underwater?” “…when the [polarizing] screen was oriented to exclude the maximum spacelight … fishes stood out in greater contrast against their background.

” “… simple polarizing screen will be less versatile than the system found in Octopus, where there is the intra-ocular ability to distinguish light polarized in one plane from that polarized in another.

” Lythgoe, 1972 (

Handbook Sensory Physiol

) : “…there is a strong possibility that it [polarization] could be useful for improving the visibility of under water.

”

distant objects

, especially

Polarization of Veiling Light

B B

• Veiling light is partially polarized Y. Schechner & N. Karpel, polarization-based recovery

Image Components

24 veiling light signal S B scattering

L

Veiling light = Spacelight = Path radiance = Backscatter Schechner, Karpel, underwater vision

Signal Polarization

• Rough surfaces : naturally depolarize • Specular reflection : weaker than in air • Multiple scattering • Signal decreases with distance / veiling-light increases At large distance: signal polarization has a negligible effect (Supported by Shashar, Sabbah & Cronin 2004) Y. Schechner & N. Karpel, polarization-based recovery

Polarization Photography

25

I

min

Past Polarization-Based Methods

I

max

I

min

I

max

I

max

I I

min min

I

max Raw images Polarization-difference imaging Degree of polarization

Model

camera 2 input images :

Recovery

camera 2 input images :

Recovery

Aqua-polaricam

Y. Schechner & N. Karpel, underwater imaging

Experiments

Experiment

Eilat, 26m underwater

I

min Best polarization image Y. Schechner & N. Karpel, underwater imaging

Naive White Balancing

26m underwater Y. Schechner & N. Karpel, underwater imaging

Y. Schechner & N. Karpel, underwater imaging 26m underwater

Range Map

Attenuation Image components backscatter Y. Schechner & N. Karpel, underwater imaging

Shape Reconstruction of Transparent Objects

Miyazaki et al •Incident light is completely unpolarized.

•Index of refraction is given.

•Exploit relation between degree of polarization and angle of incidence (Surface normal).

Relationship between DOP and Angle of Incidence Two-way ambiguity in recovered angle of incidence: Manually disambiguate, use multiple views or use prior knowledge (convex, concave, etc).

Recovered Shape

NEXT WEEK Volumetric Scattering and its Applications to Computer Vision and Computer Graphics

Lectures #18, #19, #20