Transcript Dror Baron Marco Duarte Shriram Sarvotham Michael Wakin

Compressive
Optical Imaging Systems –
Theory, Devices, Implementation
Richard Baraniuk
Kevin Kelly
David Brady
Rebecca Willett
Rice University
Duke University
Rice/Duke | Compressive Optical Devices | August 2007
Project Overview
Richard Baraniuk
Rice/Duke | Compressive Optical Devices | August 2007
Digital Revolution
camera arrays
hyperspectral cameras
distributed camera networks
Sensing by Sampling
• Long-established paradigm for digital data acquisition
– sample data at Nyquist rate (2x bandwidth)
– compress data (signal-dependent, nonlinear)
– brick wall to resolution/performance
sample
compress
sparse
wavelet
transform
receive
decompress
transmit/store
Compressive Sensing (CS)
• Directly acquire “compressed” data
• Replace samples by more general “measurements”
compressive sensing
receive
reconstruct
transmit/store
Compressive Sensing
• When data is sparse/compressible, can directly
acquire a condensed representation with
no/little information loss
through dimensionality reduction
measurements
sparse
signal
sparse
in some
basis
Compressive Sensing
• When data is sparse/compressible, can directly
acquire a condensed representation with
no/little information loss
• Random projection will work
measurements
sparse
signal
sparse
in some
basis
for signal reconstruction
[Candes-Romberg-Tao, Donoho, 2004]
Compressive Optical Imaging Systems –
Theory, Devices, and Implementation
• $400k budget for roughly April 2006-2007
– administered by ONR
– Rice portion expended; Duke portion in NCE
• Goals:
– forge collaboration between Rice and Duke teams
– demonstrate new Compressive Imaging technologies
 hardware testbeds/demos at Rice and Duke
 new theory/algorithms
– quantify performance
– articulate emerging directions
• Collaborations:
– telecons, visits, joint projects, joint papers, artwork
Gerhard Richter
4096 Farben / 4096 Colours
1974
254 cm X 254 cm
Laquer on Canvas
Catalogue Raisonné: 359
Museum Collection:
Staatliche Kunstsammlungen
Dresden (on loan)
Sales history: 11 May 2004
Christie's New York Post-War
and Contemporary Art
(Evening Sale), Lot 34
US$3,703,500
Rice/Duke | Compressive Optical Devices | August 2007
Gerhard Richter
Dresden Cathedral
Stained Glass
Rice/Duke | Compressive Optical Devices | August 2007
Agenda
• Rebecca Willett, Duke
[theory/algorithms]
• Kevin Kelly, Rice
[hardware]
• David Brady, Duke
[hardware]
• Richard Baraniuk, Rice
[theory/algorithms]
• Discussion and Conclusions
Compressive
Image Processing
Richard Baraniuk
Rice/Duke | Compressive Optical Devices | August 2007
Mike
Wakin
Matthew
Moravec
Mona
Sheikh
Jason
Laska
Marco
Duarte
Mark
Davenport
Shri
Sarvotham
Petros
Boufounos
Image
Classification/Segmentation
using
Duke Hyperspectral System
(with Rebecca Willett)
Rice/Duke | Compressive Optical Devices | August 2007
Information Scalability
• If we can reconstruct a signal from compressive
measurements, then we should be able to perform
other kinds of statistical signal processing:
– detection
– classification
– estimation …
• Hyperspectral image
classification/segmentation
Classification Example
spectrum 2
spectrum 1
spectrum 3
Nearest Projected Neighbor
• normalize
measurements
• compute nearest
neighbor
Naïve Results
block size
32
16
8
Results
naïve independent classification
tree-based classification
Voting / Cycle Spinning
16
block radius in pixels
20
28
32
24
Summary
• Direct hyperspectral classification/segmentation
without reconstructing 3D data cube
• Future directions
– replace nearest projected neighbor with more sophisticated
methods
 smashed filter
 projected SVM
 quad-tree based multiscale segmentation (HMTseg, …)
• Joint paper in the works
Performance Analysis
of
Multiplexed Cameras
Rice/Duke | Compressive Optical Devices | August 2007
Single-Pixel Camera Analysis
photon
detector
DMD
DMD
random
pattern on
DMD array
• Analyze performance in terms of
– dynamic range and #bits of A/D
– MSE due to photon counting noise
– number of measurements
image
reconstruction
or
processing
Single Pixel Image Acquisiton
For a N-pixel, K-sparse image under T-second exposure:
• Raster Scan:
Acquire one pixel at a time,
repeat N times
• Basis Scan:
Acquire one coefficient of image
in a fixed basis at a time,
repeat N times
• CS Scan:
Acquire one incoherent/random
projection of the image at a time,
repeat
times
Rice/Duke | Compressive Optical Devices | August 2007
Worst-Case Performance
•
•
•
•
•
N: Number of pixels
P: Number of photons per pixel
T: Total capture time
M: Number of measurements
CN: CS noise amplification constant
• Sensor array shown as baseline
• Table shows requirements to match worst-case
performance
• CS beats Basis Scan if
Rice/Duke | Compressive Optical Devices | August 2007
Single Pixel Camera
Experimental Performance
N = 16384
M = 1640
 = Daub-8
Rice/Duke | Compressive Optical Devices | August 2007
Multiplexed Camera Analysis
Dude, you gotta
multiplex!
S photon
detectors
lens(es)
DMD
random
pattern on
DMD array
DMD
image
reconstruction
or
processing
S-Pixel Camera Performance
•
•
•
•
•
N: Number of pixels
P: Number of photons per pixel
T: Total capture time
M: Number of measurements
CN: CS noise amplification constant
Sensor array shown as baseline
M measurements split across S sensors
Single pixel camera: S = 1
Rice/Duke | Compressive Optical Devices | August 2007
S-Pixel Camera Performance
•
•
•
•
•
N: Number of pixels
P: Number of photons per pixel
T: Total capture time
M: Number of measurements
CN: CS noise amplification constant
Sensor array shown as baseline
M measurements split across S sensors
Single pixel camera: S = 1
CS beats Basis Scan if
Rice/Duke | Compressive Optical Devices | August 2007
Smashed Filter –
Compressive Matched Filtering
Rice/Duke | Compressive Optical Devices | August 2007
Information Scalability
• If we can reconstruct a signal from compressive
measurements, then we should be able to perform
other kinds of statistical signal processing:
– detection
– classification
– estimation …
• Smashed filter:
compressive matched filter
Matched Filter
• Signal classification in additive white Gaussian noise
– LRT: classify test signal as from Class i
if it is closest to template signal i
– GLRT: when test signal can be a
transformed version of template,
use matched filter
• When signal transformations are well-behaved,
transformed templates form low-dimensional
manifolds
– GLRT
= matched filter
= nearest manifold classification
M3
Compressive LRT
• Compressive observations
• By the Johnson-Lindenstrauss Lemma,
random projection preserves pairwise distances
with high probability
Smashed Filter
• Compressive observations of transformed signal
M3
 M3
• Theorem: Structure of smooth manifolds is
preserved by random projection w.h.p. provided
distances, geodesic distances, angles, volume,
dimensionality, topology, local neighborhoods, …
[Wakin et al 2006; to appear in Foundations on Computational Mathematics]
Stable Manifold Embedding
Theorem:
Let F ½ RN be a compact K-dimensional manifold with
– condition number 1/t (curvature, self-avoiding)
– volume V
Let  be a random MxN orthoprojector with
Then with probability at least 1-r, the following
statement holds: For every pair x,y 2 F
[Wakin et al 2006]
Manifold Learning
from Compressive Measurements
ISOMAP
HLLE
Laplacian
Eigenmaps
M=15
M=20
M=15
R4096
RM
Smashed Filter – Experiments
• 3 image classes:
tank, school bus, SUV
• N = 65536 pixels
• Imaged using single-pixel CS camera with
– unknown shift
– unknown rotation
Smashed Filter – Unknown Position
• Object shifted at random (K=2 manifold)
• Noise added to measurements
• Goal:
identify most likely position for each image class
more noise
number of measurements M
classification rate (%)
avg. shift estimate error
identify most likely class using nearest-neighbor test
more noise
number of measurements M
Smashed Filter – Unknown Rotation
• Object rotated each 2 degrees
• Goals:
identify most likely rotation for each image class
• Perfect classification with
as few as 6 measurements
• Good estimates
of rotation with
under 10
measurements
avg. rot. est. error
identify most likely class using nearest-neighbor test
number of measurements M
How Low Can M Go?
• Empirical evidence that many fewer than
measurements are needed
for effective classification
• Late-breaking results
(experimental+nascent theory)
Summary – Smashed Filter
• Compressive measurements are info scalable
reconstruction > estimation > classification > detection
• Random projections preserve structure of smooth
manifolds (analogous to sparse signals)
• Smashed filter: dimension-reduced GLRT for
parametrically transformed signals
– exploits compressive measurements and manifold structure
– broadly applicable:
targets do not have to have sparse
representation in any basis
– effective for detection/classification
– number of measurements required appears to be
independent of the ambient dimension
Compressive
Phase Retrieval
for
Fourier Imagers
Rice/Duke | Compressive Optical Devices | August 2007
Coherent Diffraction Imaging
• Image by sampling in Fourier domain
• Challenge:
we observe only the magnitude
of the Fourier measurements
Phase Retrieval
• Given:
Fourier magnitude
+
additional constraints (typically support)
• Goal:
Estimate phase of Fourier transform
• Compressive Phase Retrieval (CPR)
replace image support constraint with a
sparsity/compressibility constraint
nonconvex reconstruction
Conclusions
and
Future Directions
Rice/Duke | Compressive Optical Devices | August 2007
Project Outcomes
• Forged collaboration between Rice and Duke teams
– several joint papers in progress
• Demonstrated new Compressive Imaging
technologies
– hardware testbeds/demos
 hyperspectral, low-light, infrared DMD cameras
 coded aperture spectral imagers
– new theory/algorithms
 spectral image reconstruction/classification methods
 smashed filter
• Quantified performance
– coded aperture tradeoffs
– multiplexing tradeoff
– number of measurements required for reconstruction/classification
Rice/Duke | Compressive Optical Devices | August 2007
Emerging Directions
• Nonimaging cameras
–
–
–
–
exploit information scalability
attentive/adaptive cameras
meta-analysis
separating “imaging process” from “display”
• Multiple cameras
– image beamforming, 3D geometry imaging, …
• Deeper links between physics and signal processing
– significance of coherence and spectral projections
• Links to analog-to-information program
– nonidealities as challenges vs. opportunities
• Other modalities
– THz, LWIR/MWIR, UV, soft x-rays, …
N- Pixel Camera Performance
•
•
•
•
•
N: Number of pixels
P: Number of photons per pixel
T: Total capture time
M: Number of measurements
CN: CS noise amplification constant
Sensor array shown as baseline
1 sensor per pixel - CS is unnecessary
Rice/Duke | Compressive Optical Devices | August 2007
Smashed Filter under Poisson noise
• Problem: vehicle image classification under variable
parameter (shift, rotation, etc.)
• Image acquisition: M random projections under
signal-dependent (Poisson) noise with single pixel
camera
• Limited capture time T split among M projections
• Solution: use articulation manifold structure and
generalized maximum likelihood classification
(smashed filter)
Rice/Duke | Compressive Optical Devices | August 2007
Smashed Filter performance
under Poisson noise
Shift (2D manifold)
Rotation (1D manifold)
• Small number of measurements M for good performance
• “Sweet spot” on M for shorter exposures T
Rice/Duke | Compressive Optical Devices | August 2007
CS Hallmarks
• CS changes the rules of the data acquisition game
– exploits a priori signal sparsity information
• Universal
– same random projections / hardware can be used for
any compressible signal class
(generic)
• Democratic
– each measurement carries the same amount of information
– simple encoding
– robust to measurement loss and quantization
• Asymmetrical (most processing at decoder)
• Random projections weakly encrypted
Smashed Filter:
How Low Can M Go?
Rice/Duke | Compressive Optical Devices | August 2007
Preservation of Manifold
Structure
• Manifold Learning

Used for classification, visualization of high dimensional
data, robust parameter estimation
• Network of single-pixel cameras ==== Randomly
projected version of low-dimensional image
manifold.


New result: stable manifold learning is possible without
ever reconstructing the original images
Number of measurements sufficient for arbitrarily small
learning error: linear in the information level K of the
manifold
Rice/Duke | Compressive Optical Devices | August 2007
Translating disk manifold
N = 64 x 64 = 4096, K = 2
• Learning algorithm: LTSA (Zhang, Zha. 2004.)
25 random
projections
50 random
projections
Learning with original data:
(N = 4096)
Rice/Duke | Compressive Optical Devices | August 2007
100 random projections
Manifold learning using random
projections
• Demonstrates that random projections contain
sufficient information about the manifold structure
• Two stages in manifold learning
– Intrinsic dimension estimation
– Construction of nonlinear map into low-dimensional
Euclidean space
– New result: estimation errors in both stages due to
dimensionality-reducing projections can be controlled up
to arbitrary accuracy with small number of
measurements
• Ideal for distributed networks; sensors need to
transmit very few pieces of information to the
centralized learning algorithm
Rice/Duke | Compressive Optical Devices | August 2007
Intrinsic Dimension estimation
• GP algorithm used directly on random projections
of hyperspheres
• Empirically compute the number of measurements
required for estimate to be within 10% of the
original.
• Observation: M linear in the intrinsic dimension K
Rice/Duke | Compressive Optical Devices | August 2007
Real data: Hand rotation
database
N = 64 x 60 = 3840, K = 2
Rice/Duke | Compressive Optical Devices | August 2007
New Bound for Classification?
• Smashed Filter – Nearest Neighbor classifier
• Indyk, Naor. 2007 : preservation of approximate
nearest neighbors requires merely O(K) random
projections
2
M
=
O
K
log
2/
D
/
D
• Minimum number of measurements required for
classification in noiseless case (where D is the
minimum separation between signal classes ):
Rice/Duke | Compressive Optical Devices | August 2007
Experiment: Hyperspherical
manifolds
• 1000 labeled training samples each from two unit
3-dimensional hyperspheres, separated by a
distance D along an arbitrary direction in 2000dimensional space
• Generate unlabeled samples, perform nearest
neighbor classification in the compressed
(“smashed”) domain
• Determine minimum number of measurements M
required to obtain 99% classification rate.
• Bound: M decreases as square of the separation
distance.
Rice/Duke | Compressive Optical Devices | August 2007
Hyperspherical manifolds:
empirical verification of bound
Rice/Duke | Compressive Optical Devices | August 2007
Why Does CS Work (1)?
• Random projection not full rank, but stably embeds
– sparse/compressible signal models (CS)
– point clouds (JL)
into lower dimensional space with high probability
• Stable embedding: preserves structure
– distances between points, angles between vectors, …
Why Does CS Work (1)?
• Random projection not full rank, but stably embeds
– sparse/compressible signal models (CS)
– point clouds (JL)
into lower dimensional space with high probability
• Stable embedding: preserves structure
– distances between points, angles between vectors, …
provided M is large enough: Compressive Sensing
K-sparse
model
K-dim planes
CS Signal Recovery
• Recover sparse/compressible signal x from CS
measurements y via linear programming
K-sparse
model
K-dim planes
recovery
Why Does CS Work (2)?
• Random projection not full rank, but stably embeds
– sparse/compressible signal models (CS)
– point clouds (JL)
into lower dimensional space with high probability
• Stable embedding: preserves structure
– distances between points, angles between vectors, …
provided M is large enough: Johnson-Lindenstrauss
Q points
Tree-based classification
• Refine classification of blocks having neighbors
from a different class
Tree-based classification
• Refine classification of blocks having neighbors
from a different class
Tree-based classification
• Refine classification of blocks having neighbors
from a different class