Transcript CS-MUVI Video compressive sensing for spatial multiplexing

CS-MUVI
Video compressive sensing for
spatial multiplexing cameras
Aswin Sankaranarayanan, Christoph Studer, Richard G. Baraniuk
Rice University
Single pixel camera
Photo-detector
Digital
micro-mirror
device
Single pixel camera
• Each configuration of
micro-mirrors yield ONE
compressive
measurement
• Non-visible
wavelengths
• Sensor material costly
in IR/UV bands
• Light throughput
Photo-detector
Digital
micro-mirror
device
• Half the light in the scene is directed to the
photo-detector
• Much higher SNR as compared to traditional
sensors
Single pixel camera
• Each configuration of
micro-mirrors yield ONE
compressive
measurement
Photo-detector
• static scene assumption
• Key question:
Can we ignore motion
in the scene ?
Digital
micro-mirror
device
SPC on a time-varying scene
time varying
scene
measurements
t=1
t=W
compressive
recovery
• Naïve approach:
Collect W measurements together
to compute an estimate of an image
what happens ?
SPC on a time-varying scene
t=1
t=W (small)
Small W
Less motion blur
Lower spatial resolution
t=W (large)
Large W
Higher spatial resolution
More motion blur
Tradeoff
Temporal resolution vs. spatial resolution
SPC on a time-varying scene
sweet
spot
Lower spatial res.
Higher temporal res.
Higher spatial res.
Lower temporal res.
Dealing with Motion
• Motion information can help in obtaining better
tradeoffs [Reddy et al. 2011]
– State-of-the-art video compression
Dealing with Motion
• Motion information can help in obtaining better
tradeoffs [Reddy et al. 2011]
– State-of-the-art video compression
naïve reconstruction
motion estimates
Key points
• Motion blur and the failure of the sparsity
assumption
– Use least squares recovery ?
• Recover scene at lower spatial resolution
– Lower dimensional problem requires lesser number of
measurements
– Tradeoff spatial resolution for temporal resolution
• Least squares and random matrices
– Random matrices are ill-conditioned
– Noise amplification
• Hadamard matrices
– Orthogonal (no noise amplification)
– Maximum light throughput
– Optimal for least squares recovery
[Harwit and Sloane, 1979]
Hadamard + least sq. recovery
Hadamard
Random
Hadamard + least sq. recovery
Designing measurement matrices
• Hadamard matrices
– Higher temporal resolution
– Poor spatial resolution
• Random matrices
– Guarantees successful l1 recovery
– Full spatial resolution
• Can we simultaneously have both properties
in the same measurement matrix ?
Dual-scale sensing (DSS) matrices
Key Idea: Constructing high-resolution measurement
matrices that have good properties when downsampled
1. Start with a
row of the
Hadamard
matrix
2. Upsample
3. Add
high-freq.
component
CS-MUVI: Algorithm outline
1. obtain
measurements
with
DSS matrices
2. lowresolution
initial estimate
3. motion
estimation
4. compressive
recovery with
motion
constraints
t=1
t=W
t=t0
t=t0+W
t=T
Simulation result
CS-MUVI on SPC
Single pixel camera setup
InGaAs Photo-detector (Short-wave IR)
SPC sampling rate: 10,000 sample/s
Number of compressive measurements: M = 16,384
Recovered video: N = 128 x 128 x 61 = 61*M
Object
CS-MUVI: IR spectrum
initial estimate
Upsampled
Recovered Video
Joint work with Xu and Kelly
CS-MUVI on SPC
Naïve frame-to-frame recovery
CS-MUVI
Joint work with Xu and Kelly
CS-MUVI summary
• Key ingredients
– Novel Measurement matrix design
– Exploiting state-of-the-art motion model
• One of first practical video recovery algorithm
for the SMC
dsp.rice.edu
CS-MUVI summary
• Limitations
– Need a priori knowledge of object speed
– Motion at low-resolution
– Robustness to errors in motion estimates
• Future work
– Dual-scale to multi-scale matrix constructions
– Multi-frame optical flow
– Online recovery algorithms
dsp.rice.edu