Transcript Imager Design using Object-Space Prior Knowledge M. A. Neifeld University of Arizona OUTLINE 1.

Imager Design using Object-Space Prior Knowledge
M. A. Neifeld
University of Arizona
OUTLINE
1. The Last Slot
2. Introduction
3. PSF Engineering
4. Feature-Specific Imaging
Neifeld IMA 2005
Neifeld IMA 2005
Introduction: objects are not iid pixels.
- Conventional cameras are designed to image iid pixels
 impulse-like point-spread-functions (identity transformation)
 generic metrics such as resolution, field of view, SNR, etc.
- Real objects are not iid pixels so don’t estimate pixels
- This keeps the compression guys employed!
- (106 pixels)(3 colors/pixel)(8 bits/color) = 2.4x107 bits
- (1011 people)(4x109 years)(109 images/year) = 4x1029 images  <100 bits
- The set of “interesting” objects is small
- Many ways to characterize “interesting” objects: power spectra, principal
components, Markov fields, wavelet projections, templates, task-specific
models, finite alphabets, etc.
Information depends upon task:
 Option 1 - this is a random image  I = 107 bits
 Option 2 – this is a “battlefield” image  I = ? bits
… how to quantify PDF!
 Option 3 – this image either contains a tank or not  I = 1bit
… task-specific source model
Neifeld IMA 2005
Introduction: post-processing exploits priors.
- Linear Restoration: de-noising and de-blurring exploit noise statistics, object power
spectra, principal components, wavelets, …
- Nonlinear Restoration: super-resolution uses finite support, positivity, finite alphabet,
power spectra, wavelets, principal components, isolated points, …
- Recognition: features, templates, image libraries, syntax, invariance, …
- Finite Alphabet Post-Processing Examples
LADAR
Signal Reconstructed using Largest Return at SNR = 10 dB
Signal Reconstructed Ideally
5
10
10
15
15
35
20
Scan # 
Scan # 
30
Object
25
largest return
rmse = 7.3mm
5
20
25
30
35
40
40
45
45
50
5
10
15
20
25 using30Wiener Filter
35 at SNR
40 = 10 dB
45
Signal
Reconstructed
Range Information (scaled as intensity values) 
50
55
55
5
35
40
45
10
15
20
Scan # 
Scan # 
30
Signal
using
Viterbi
20 Reconstructed
25
30
35 at SNR40= 10 dB 45
50
IBP – 28%
55
25
30
35
40
45
50
50
55
55
5
10
15
20
25
30
35
40
Received Range Information (scaled as intensity values) 
45
50
55
Viterbi
rmse = 0.6mm
25
15
Received Range Information (scaled as intensity values) 
Wiener
rmse = 5.8mm
20
10
Measurement
5
5
15
Object
50
55
10
Multi-Frame Super-Resolution
5
10
15
IBPP – 24%
20
25
30
35
40
Received Range Information (scaled as intensity values) 
45
50
55
Axial extent of target = Temporal pulse width = 30mm.
Target feature size = Scan step size = 4.6mm
2D4 - 2%
Optical blur = 1.5 and pixel-blur = 2.
Reconstruction from 2 images, σ = 1%
Neifeld IMA 2005
Introduction: plausibility of a single pixel imager.

Measure only what you want to know
Source
volume
 Fluorescent markers
 Distant “bright” objects: aircraft, missile, stars
Imager
r1
r2
y
rM
x
M : Number of point sources
z
Strong Object Model:

Equal-intensity monochromatic point sources

Scene is completely specified by sources positions: r1 r2 … rM
Imager Goals:
Estimate point source position(s): { r1 r2 … rM }
 Conventional image may be formed as a post-processing step

r1 r2 … rM
Conventional
image
Neifeld IMA 2005
Introduction: information-based design.
 Optimize imager based on information metric.
 Maximize measurement entropy.
 Select detector sizes and positions based on measurement pdf.
1
Source Volume
phase
mask
40cm
2
2
1
3
3
1cm3
source power = 0.5mW
Measurement log-pdf
Lens
Measurement log-pdf
Detector NEP=2nW
d1 ,h1
Measurement log-pdf
cubic phase
random phase
Neifeld IMA 2005
Introduction: single pixel imager results.
Single Source in Volume
 Detector(s) : Imager Type 
Multiple Sources in Volume
Conventional
CPM
RPM
One detector in one aperture
21%
39%
65%
Two detectors in one aperture
30%
54%
74%
Two detectors in two apertures
36%
74%
89%
 Object-space prior knowledge should inform the optical design
 Let’s utilize this viewpoint in a more useful problem domain
Neifeld IMA 2005
PSF ENGINEERING
Neifeld IMA 2005
PSF Engineering: Under-Sampled Imagers
 Imagers for which pixel size > optical spot size. .
 Large pixels result in under-sampling/aliasing.
 Sub-pixel shifted measurements to resolve ambiguity.
shift camera
Frame 1
spatial ambiguity
…..
Frame 2
 Optical degrees of freedom not exploited.
 We consider engineering optical point spread function.
Frame K
Neifeld IMA 2005
Imaging Model
Object: f
N = 512x512
Imaging operator: H Measurements: g
Phase-mask
Optics details:
 Resolution = 0.2mrad/1mm
 Field of view = 0.1 rad
 Thickness = 5mm
 Aperture = 2.75mm
 F/# = 1/1.8
Sub-pixel shifts
Sensor details:
 Pixel = 7.5 mm
 Under-sampling = 15x
 Full well capacity = 49ke Spectral bandwidth = 10nm
 Center wavelength = 550nm
M = 34x34
 Single frame signal to noise ratio: SNR = 10log[sqrt(Ne)] = 23.3dB
 SNR can be improved via multi-frame averaging ~ sqrt(K)
 Total photon-count is kept constant over multiple-frames.
Neifeld IMA 2005
Linear Reconstruction
 Linear imaging model: g = Hf + n (note: n is AWGN)
 Block-wise shift-invariant imaging operator H is M x N
 Problem: M << N (e.g., M=N/15)
^
 Linear minimum mean square error (LMMSE) reconstruction: f = Wg
 LMMSE operator: W = RfHt(HRfHt+Rn)-1
 No Priors = flat PSD
 Priors = power law PSD or triangle PSD
Example training objects
Power Law PSD(f) = 1/f
PSD model
Neifeld IMA 2005
Performance Measures
 Root Mean Squared Error:
RMSE 
100 
Object
Composite
Channel
Hc
 Angular resolution:
g
n

 x
  arg min  sinc 2 

 


g
+

 


fˆ 

2
[%]
LMMSE
Reconstruction
+
Composite
Channel
Hc
Point Object
f = d(r)
 f  fˆ 
255
n
RMSE=8.6%
2
Reconstruction to
Diffraction-limited
sinc2
=0.4mrad
^
f
Neifeld IMA 2005
Conventional/TOMBO Imager Results
TOMBO Imager
Conventional Imager
Shift-sensor
RMSE for TOMBO
sub-pixel shift
Sub-pixel shifted
measurements
Resolution for TOMBO
Neifeld IMA 2005
Alternate PSF
 Consider use of extended point spread function(PSF)
impulse-like PSF
 Design issue #1: retain full optical bandwidth
 Design issue #2: tradeoff SNR for condition number
 Pseudo-Random Phase masks for extended PSF
extended PSF
Realization of a spatial Gaussian random process.
 x2 

R x       exp   2  ,    

 4 
 - mask roughness
 - mask correlation length
Pseudo-Random Phase mask Enhanced Lens (PRPEL)
Example PSF(=0.5,=10 )
Modulation Transfer Function
Neifeld IMA 2005
Resolution Results
Resolution for PRPEL and TOMBO
All designs use optimal roughness.
 Note more rapid convergence of
PRPEL compared to TOMBO.
 Higher resolution achieved by
PRPEL at reduced number of
frames.
 PRPEL achieves 0.3mrad
resolution at K=5 compared to
K=12 for TOMBO.
Neifeld IMA 2005
RMSE Results
RMSE for PRPEL and TOMBO
 PRPEL makes effective use of prior knowledge at K=1
 Note more rapid convergence of PRPEL.
 PRPEL consistently out-performs TOMBO.
TOMBO
PRPEL
K=1
K=1
K=2
K=2
K=3
K=3
Neifeld IMA 2005
PRPEL Summary
4% RMSE requirement
RMSE achieved at M=N/4
Imager Type
Number of Frames
TOMBO
PRPEL
Imager Type 
(K=4)
TOMBO
PRPEL
K
5
4
RMSE
4.2%
3.9%
0.3mrad Resolution requirement
Resolution achieved at M=N/4
Imager Type
Number of Frames
TOMBO
PRPEL
Imager Type 
(K=4)
TOMBO
PRPEL
K
12
5
Resolution
0.60mrad
0.35mrad
 PRPEL imager achieves 60% improvement in resolution.
 PRPEL imager obtains 22% improvement in RMSE.
Neifeld IMA 2005
PSF Engineering via SPEL
 Sine-Phase mask Enhanced Lens(SPEL) :
 ( x)   i sin  i x   i 
N
i 1
Phase offset
Spatial-frequency
Amplitude


Phase-mask
 Pick N=3: yields 12 free parameters for optimization.
 Optimization criteria: RMSE 
100 

f  fˆ

2
[%]
255
 RMSE computed over object class using LMMSE operator.
 PSF is optimized for each value of K.
Neifeld IMA 2005
Optimized PSF
K=1
Observations

Note smaller support of SPEL
PSF compared to PRPEL PSF.
 SPEL PSF also contains subpixel structure.
 SPEL PSF has more efficient
photon-distribution.
K=2
Observations

PSF support reduces with
increasing K.

SPEL PSF is array of delta
pulses.
Neifeld IMA 2005
Optimized PSF: System Implications
K=16
Observations

SPEL PSF converges to delta
pulses as K increases.

In limit K16 we observe that
SPEL PSF to converge to
TOMBO-like PSF.
Neifeld IMA 2005
Results
RMSE : Power law PSD
PRPEL
SPEL
K=1
K=1
K=2
K=2
K=3
K=3
RMSE for SPEL, PRPEL, and TOMBO
 SPEL provides best use of prior knowledge for K=1
 SPEL outperforms TOMBO by 47% in terms of RMSE(K=8).
 SPEL improves RMSE by 35% compared to PRPEL (K=8).
Neifeld IMA 2005
Results
Angular resolution
Resolution for SPEL,PRPEL and TOMBO
 Note PSF optimization was
performed over RMSE.
 SPEL out-performs TOMBO.
 SPEL performance compared to
PRPEL improves with increasing K.
 PSF engineering can exploit weak object prior knowledge to improve performance
 Stronger object prior knowledge can enable non-traditional image measurement
Neifeld IMA 2005
FEATURE-SPECIFIC IMAGING
Neifeld IMA 2005
Passive Feature-Specific Imaging: Motivation
Conventional imaging system
Feature
extraction
PCA, ICA, Fisher,
Wavelet, etc.
Features
Task
Restoration, recognition,
compression, etc.
noisy image
noise
Feature-specific
optics
Feature-specific imaging
system (FSI)
Features
Task
noise
 Feature-Specific Imaging (FSI) is a way of directly measuring linear features
(linear combinations of object pixels).
 Attractive solution for tasks that require linear projections of object space
 Let’s consider a case for which task = pretty picture
Neifeld IMA 2005
FSI for Reconstruction
 PCA features provide optimal measurements in the absence of noise
Noise-free reconstruction:
y  Fx
min
xˆ  My
Fpca
  E{|| x  xˆ ||2 }
m
subject to ||F||1  max{ | fij |}  1
i
PCA solution :
j 1
photon count constraint
Result using PCA features:
T
M pca = Fpca
General solution :
F  AFpca
A is any invertable matrix
M general  RxF(FRxFT )1
Neifeld IMA 2005
Optimal Features in Noise
 PCA features are not optimal in presence of noise
Noise-free problem statement:
y  Fx  n
y  Fx
xˆ  My
xˆ  My
Mopt  RxFT (FRxFT   2I)-1 Wiener - operator
2
ˆ
2 T E{|| x  T
min

x
||
  Tr{FR xF (FR xF  } 2 I)-1}  Tr{R x }
• Object block size = 4x4
• Noise = AWGN
• We use stochastic tunneling
to optimize/search
m
subject to ||F||1  max{ | fij |}  1
i
E{|| x ||2 }
SNR  10 log(
)
2

j 1
F
Note: PCA error is no longer
monotonic in the number of
features  trade-off
between truncation error
and photon count constraint
RMSE = 12.9
RMSE = 124
RMSE = 12
RMSE = 11.8
Fpca
|| Fpca ||1
Fopt
Neifeld IMA 2005
Optimal Features in Noise
 Error increases as number of feature increases for PCA solution
 Reconstructed is improved significantly by using optimal solution
 Optical implementation requires non-negative projections
Neifeld IMA 2005
Passive FSI Result Summary
 Optimal FSI is always superior to conventional imaging
 Non-negative solution is a good experimental system candidate
Neifeld IMA 2005
Passive FSI for Face Recognition
•
Face recognition from grayscale image feature measurements
•
Class of 10 faces, 600 images per face
•
Training = 3000 faces and testing = 3000 faces
•
Features: wavelet, PCA, Fisher, …
•
Recognition algorithms:
- k – nearest neighbor based on Euclidean distance metric
- 2-layer neural networks batch trained using back-propagation with momentum
Comparison of PCA recognition with AWGN
Sample images from face database
[Each image is 128x96]
Recognition performance [%]
100
90
FSI
80
70
60
50
40
0 mux
0 conv
0_1 mux
0_1 conv
30
20
Conventional
10
0
First Wavelet feature of the above images
[Each feature is 8x6]
0
250
500
750
1000
1250
AWGN standard deviation
1500
1750
2000
Neifeld IMA 2005
Passive FSI Optical Implementations
Neifeld IMA 2005
Active Feature-Specific Imaging: Motivation
• What is active illumination ?
Object
• Project known structure onto scene
• Additional degrees of freedom
improve imager performance
Illumination
pattern
Projector
• Past work on active illumination focused on:
• Obtain depth-information for 3D objects
• Enhanced resolution for 2D objects
• Our goals:
• Improve object- and/or task-specific performance
• Simplify light collection hardware
Conventional
cameras
Neifeld IMA 2005
FSAI System Flow Diagram
• Illumination patterns are eigenvectors (refer as PCA - FSAI)
16 × 16 replication
of eigenvector P1
Light Collection
Object G
Sequence of
illumination patterns
PM
P2
16 × 16
detector
ri  [H][diag (Pi )]G  ni
64 × 64
[diag (Pi )]G
H (optics operator)
• Advantages
Photodetector
noise (AWGN)
[H][diag (P )]G
i
~ ̂ i
(Estimate of feature weight)
• Small number of detectors
• High measurement SNR
• Task is to produce object estimate using these values
ece
 r1 
r 
 2
R . 
 
 . 
 rM 
Vector of
Measurements
Neifeld IMA 2005
FSAI Post-Processing
 r1 
r 
Measurement
 2
R . 
vector
 
 . 
 rM 
Linear postprocessing
Ŵ
Ĝ=ŴR ≠
̂
i
Pi
(suboptimal in noise)
• Post-processing operator Ŵ is obtained by minimizing J
J  E{trace[(G  Wˆ R)(G  Wˆ R) T ]}
(mean square error )
• The MMSE operator is given by:
~T ~
~T
ˆ R H
W
[
H
R
H
 R n ] 1 ,
G
G
~
~
where H  [ H i, j ] M  N 2 and
~
H i, j 
N2
 [H][diag (P )]
i
n 1
n, j
N 2 = number of pixels,
M = number of patterns
and R G  object correlatio n matrix, R n  noise covariance matrix.
• Metric to evaluate reconstructions :
number of objects
1
1
RMSE 

number of objects
N2
k 1
ece
N2
 (G
i 1
ki
ˆ )2
G
ki
Neifeld IMA 2005
Illumination Using Optimal Patterns
• PCA vectors are not optimal in presence of noise
J  E{trace[(G  Wˆ R)(G  Wˆ R) T ]} ( R contains noise )

J PCA which is | G  Gˆ | with Gˆ 
2
K
α P
i 1
i
i
• Minimize the residual MMSE (JMMSE) with respect to both Pi’s and Ti’s
~
J MMSE ( P1 ,...PM , T1 ,....TM )  Trace{RG  Wˆ H RG }
where Wˆ ( P1 ,.....PM , T1 ,.....TM )
1
2

T  ~
T
 2  2

~
~

 RG H ( P1 ...PM )  H ( P1 ...PM ) RG H ( P1 ...PM )  diag  2 , 2 ,.... 2 
TM 
 T1 T2


 
 

• Optimal features depend on M, SNR
SNR = 26 dB
PCA
M=4
optimal
PCA
M=8
optimal
ece
Neifeld IMA 2005
FSAI Results
SNR = 26 dB (LOW NOISE)
Original object
0.2
Average RMSE (LOG SCALE)
Uniform illumination
PCA-FSAI
(uniform T)
PCA-FSAI
(optimal T)
Optimal FSAI
M=4
PCA-FSAI
(u n i f o r m T)
0.1
M=8
Optimal
features
0.04
0
2
PCA - FSAI
(n o n-u n i f o r m T)
2
6
8
10 12
Number of features
ece
• Minimum from PCA-FSAI
RMSE = 0.0633
14
16
• Minimum from optimal FSAI
RMSE = 0.0465
Neifeld IMA 2005
FSAI Results Summary
Algorithm
Uniform illumination
SNR = 26 dB
SNR = 16 dB
0.067 (M = 1)
0.151 (M =1)
0.063 (M = 4)
0.0768 (M = 2)
0.063 (M > 4)
0.0768 (M > 2 )
0.0465 (M = 16)
0.07
31 %
54 %
PCA – FSAI (uniform T)
PCA – FSAI (nonuniform T)
Optimal features
Improvement of optimal FSAI
compared to uniform illumination
ece
(M = 16)
Neifeld IMA 2005
Conclusions
 Objects are not iid pixels
Pixel-fidelity should not be the goal of an imager
Need new non-traditional design metrics
 Design should reflect prior knowledge of objects
Object-specific imagers (e.g., SPEL)
Joint design of optics and post-processing
 Design should reflect prior knowledge of application
Task-specific imagers (e.g., FSI)