Transcript NIR-VIS - University of Oulu

Face image mapping from NIR to VIS
Jie Chen
Machine Vision Group
http://www.ee.oulu.fi/mvg
MACHINE VISION GROUP
Outline
•
Problem
•
Methods
•
Preliminary results
•
Plans for next period
MACHINE VISION GROUP
Face image mapping from NIR to VIS
•
Problem
– NIR: Near infrared imaging
– VIS: Visual light imaging
MACHINE VISION GROUP
Face image mapping from NIR to VIS
•
Problem
– NIR: Near infrared imaging
– VIS: Visual light imaging
MACHINE VISION GROUP
Algorithm: Patches mapping Training
•
wo
Training
ho
hp
wf
wp
hf
MACHINE VISION GROUP
Algorithm: Patches mapping Training
•
Mapping
φi,j
i1,, kj
MACHINE VISION GROUP
Looking up
Ddictionary of
face patches
and their LBP
histograms
Look up the KNN
i2,1
,j
i1,1,j
i2,0
,j
i3, j
0
1
 i4, j
K 1
i1,0
,j
i2,, jK 1
i1,, jK 1

3
i, j A patch of an input sample in S3
i1,,kj
k
k-th nearest patch in S1
 i4, j
A patch of an input sample in S4
i2,,jk
Corresponding patch of
Weight of k-th nearest neighbor
MACHINE VISION GROUP
i1,,kj in S2
Weight computing
L
( H1 , H 2 )   min( H1,i , H 2,i )
i 1
k 
k
Ddictionary of
face patches
and their LBP
histograms
K 1

p 0
Looking up
p
i3, j  ki2,, jk
MACHINE VISION GROUP
Experiments
•
Setup
– both S1 and S2 is composed of 300
samples.
• 50 subjects,
• each subject has 6 images but in
different expression (anger, disgust,
fear, happiness, sadness, and
surprise).
– wf =64, hf =80, wp=16, hp=16,
wo =12, ho =12
– Testing:using leave-one-out and K=15.
wo
ho
hp
wf
wp
hf
MACHINE VISION GROUP
Reconstructed images
(a) Input images in NIR
24.88
23.76
27.55
26.29
26.48
27.43
21.54
21.18
(b) Reconstructed images in VIS using LBP(8,1) and the PSNR
31.89
32.11
32.11
34.41
32.11
31.68
32.18
31.08
(c) Reconstructed images in VIS using the combined Multi-resolution LBP and their PSNR
MACHINE VISION GROUP
(d) Ground truth in VIS
Multi-resolution LBP (MLBP)
(P=4,R=1)
(P=8,R=1)
29
9
4
1
42
29
2
1
55
15
6
1
0
0
(P=12,R=1.5)
0
1
0
0
2
(P=16,R=2)
4
1
8
16
8
32
64 128
32
MACHINE VISION GROUP
0
0
0
0
(P=24,R=3)
0
LBP=1+8+32=41
PNSR
Pixel wise
 MAX I2
PSNR  10 log10 
 MSE

 MAX I 

20
log

10 

MSE



1 w 1 h 1
MSE 
I (i, j )  I (i, j )

wh i  0 j  0
2
LBP
MSE  1  (H1 , H2 ) 
2
 MAX I2
PSNR  10 log10 
 MSE

 1 
  20 log10 

 MSE 

MACHINE VISION GROUP
Multi-resolution LBP
L
( H1 , H 2 )   min( H1,i , H 2,i )
Looking up
i 1
 p , s  f ( p , s , c )
k 
k
Ddictionary of
face patches
and their LBP
histograms
C 1
K 1
 p , s  f ( p , s , c )    p , s , c
p
c 0
p 0
i3, j  ki2,, jk
MACHINE VISION GROUP
PSNR on MLBP
PSNR for combining multi-resolution LBP by different methods
40
35. 13
35
29. 77
30
29. 23 29. 29 29. 77 29. 64
26. 45
27. 92
24. 45
25
20
15
10
5
0
0
2
4
6
8
10
1
1- -LB
LB P
PCS
Su
Pr m
od
uc
t
M
ax
M
M in
ed
ia
Co
n
nc M
at ea
en n
at
io
n
PC
A
PSNR
CSLBP
26. 52
MACHINE VISION GROUP
Plans for next period
•
Training data:
– Use more samples (192*10 from CASIA, a group in Beijing, China)
•
Methods:
– Combine the methods proposed in the paper (A. Hertzmann,
SIGGRAPH, 2001) for better performance
MACHINE VISION GROUP