Transcript Slide 1

c[n]
Beyond Bandlimited Sampling:
Nonideal Sampling, Smoothness and
Sparsity
Yonina Eldar
Department of Electrical Engineering
Technion-Israel Institute of Technology
http://www.ee.technion.ac.il/people/YoninaEldar/
[email protected]
Rice, April 2008
c[n]
Sampling: “Analog Girl in a Digital World…”
Judy Gorman 99
Analog world
Digital world
Sampling
A2D
c[n]
Signal processing
Denoising
Image analysis …
Reconstruction
D2A
(Interpolation)
2
Introduction
Input Signals
Nonlinearities
Reconstruction
Main Problem
Can we reconstruct x(t) from c[n]?
No! Unless we know something about x(t)
x(t)
x(t) bandlimited
piece-wise linear
Different priors lead to different reconstructions
3
Introduction
Input Signals
Nonlinearities
Reconstruction
Our Point-Of-View
The field of sampling was traditionally associated with methods
implemented either in the frequency domain, or in the time domain
Sampling can be viewed in a broader sense of projection onto any
subspace or union of subspaces
Can choose the subspaces to yield interesting new possibilities (below
Nyquist sampling of sparse signals, pointwise samples of non
bandlimited signals, perfect compensation of nonlinear effects …)
4
Introduction
Nonlinearities
Input Signals
Reconstruction
Bandlimited Sampling Theorems
Cauchy (1841):
n
x(t )
x 
N
n  
Periodic & N-BL
t
N
N 1
n
x(t )   x  h(t  n N )
n 0  N 
sin( t )
h(t ) 
N sin( t / N )
H
Whittaker (1915) - Shannon (1948):
1
x ( n)
x(t )
π-BL
tn


  (t  n)


x(t ) 

 x(n)h(t  n)
n 
h(t )  sinc( t )
n 
Extensions focusing primarily on bandlimited signals with
nonuniform grids
A. J. Jerri, “The Shannon sampling theorem - its various extensions and applications:
A tutorial review”, Proc. IEEE, pp. 1565-1595, Nov. 1977.
5
Introduction
Input Signals
Nonlinearities
Reconstruction
Towards More Robust DSPs …
Limitations of standard sampling theorems:
Input bandlimited
Ideal sampling c[n]  x(tn )
Ideal reconstruction (ideal LPF)
Impractical
Towards more robust DSPs:
General inputs
Nonideal sampling: general pre-filters, nonlinear distortions
Simple interpolation kernels
6
Introduction
Nonlinearities
Input Signals
Reconstruction
Beyond Bandlimited
Two key ideas in bandlimited sampling:
Avoid aliasing
Fourier domain analysis
a(t )
Misleading concepts!
Suppose that x(t ) 

 c(n)a(t  n)
n 
with
Signal is clearly not bandlimited
Aliasing in frequency and time
Perfect reconstruction possible from samples x(n)
x[ n ]

H (e j )
a(t )
x(t )

  (t  n)
n 
Aliasing is not the issue …
7
Introduction
Nonlinearities
Input Signals
Reconstruction
Fourier Domain Can Be Misleading
Original + Initial guess
Nonideal sampling
Nonlinear distortion
x(t )
linear distortion
s(t )
c[n]
t=n
Reconstructed signal
Replace Fourier analysis by functional analysis, Hilbert space
algebra, and convex optimization
8
Introduction
Nonlinearities
Input Signals
Reconstruction
Moving To An Abstract Hilbert Space
What classes of inputs would we like to treat?
Subspace prioir
X(f )
x(t )
Bandlimited
Shift invariant subspace: x(t ) 

x(t )
Spline spaces
 c[n]a(t  n)
n 
General subspace in a Hilbert space
Smoothness constraints: || Lx ||2  U
Sparse vector model (compressed sensing): || Lx ||0  U
Analog Version?
x
9
Introduction
Input Signals
Nonlinearities
Reconstruction
Broad Sampling Framework
Beyond Ideal Bandlimited Sampling
Broad class of input signals: Subspace priors, smoothness priors
Sparse analog signals: Signals restricted to bands
Analog sampling + compressed sensing
General pre-filters
Perfect compensation of nonlinear distortions
Nonideal reconstruction filters
10
Introduction
Input Signals
Nonlinearities
Reconstruction
Outline
1. Input Signals:
Subspace methods: Perfect recovery from linear generalized samples
Smoothness priors: Minimax approximations
Sparsity priors:
Brief overview of compressed sensing (CS) for finite vectors
CS of analog signals: Blind sampling of multiband signals
2. Nonlinearties: Perfect recovery in the presence of nonlinear distortions
3. Nonideal reconstruction: Minimax approximation with simple kernels
11
Introduction
Nonlinearities
Input Signals
Reconstruction
Non-Ideal Linear Sampling
Non- ideal sampling
x(t )
Sampling functions
c[n]  x(t ), s(t  n)
s(-t)
(cn = x(t ), sn (t ) )
t=nT
(Riesz: Any linear and bounded acquisition)
Generalized antialiasing filter
Examples:
Local averaging
nT 
c[n] 

x(t )dt
Electrical circuit
x(t )
s (t )
R
nT
0
C
t  nT
c[n]
s (t )
Δ


Sampling space S: f (t )   a[n]sn (t )   a[n]s (t  nT )
n 
n 
In the sequel: T=1
12
A


S (  2 k )  B or
2
Introduction
k 


k 
S (  2 k )  0
2
Nonlinearities
Input Signals
Reconstruction
Perfect Reconstruction
Given s(t) which signals can be perfectly reconstructed?
Key observation:
x(t )
sampling space
s(t )
c[n]
tn

x(t )
PS
s(t )
c[n]
tn
x(t ), s(t  n)  x(t ), PS s(t  n)  PS x(t ), s(t  n)
Knowing c[n] is equivalent to knowing PS x(t ) if s(t  n) is a frame
(Riesz basis):
Sd (e )  F  s(t ), s(t  n)
j
c[n]

   | S (  2 k ) |
2

k 
1
Sd (ei )

s (t )
A  S d (e j )  B or S d (e j )  0
PS x(t )

  (t  n)
n 
If x(t) is in S then PS x(t )  x(t ) and perfect reconstruction is possible
13
A



S (  2 k )  B or
2
Introduction
k 

k 
S (  2 k )  0
2
Nonlinearities
Input Signals
Reconstruction
Shannon Revisited
Perfect reconstruction scheme:
x(t )
s(t )
1
Sd (ei )
c[n]
tn

s (t )
PS x(t )

  (t  n)
n 
Sd (e )  F  s (t ), s(t  n)
j
x ( n)
tn
   | S (  2 k ) |
2
k 
s (t )  sinc(t )  S d (e j )  1
Bandlimited sampling:
x(t )



  (t  n)
1


x(t )
n 
14
Introduction
Nonlinearities
Input Signals
Reconstruction
Mismatched Sampling
What if x(t) lies in a subspace A  S where A is generated by a(t) ?
If x(t )  A S  then PR impossible since x(t ), s(t  n)  0

If L2  A  S then PR possible
Ad (e )  F  a(t ), s (t  n)  
j

 A(  2 k )S
*
(  2 k )
k 
Ad (e j )  0  L2  A  S  (Christansen and Eldar, 2005)
c[n]
1
Ad (e j )

x(t )
a(t )

  (t  n)
n 
Perfect Reconstruction in a Subspace:
x(t )  A
s(t )
c[n]
tn
1
Ad (e j )

a(t )
x(t )

  (t  n)
n 
15
Introduction
Nonlinearities
Input Signals
Reconstruction
Examples
Point-wise sampling of x(t )   d[n]a(t  n):
c[n]=x[n] corresponding to s(t)=δ(t)
Can recover x(t) as long as F{a[n]}  0 (Unser and Aldroubi 94)
Bandlimited sampling:
R
s(t )
x(t )
C
1
s(t )  sinc(t )
c[n]
tn
t
a(t )  e  u (t )
50

|AF()|
|SF()|
Can x(t) be recovered
even though it is not
bandlimited?
dB
0
-50
-100
0
5
10
15
20
Frequency [rad/s]
25
30
35
16
Introduction
Nonlinearities
Input Signals
Reconstruction
Perfect Recovery
1. Compute convolutional inverse of h[n]  s(t )  a(t ) t n
h-1[n]
n0
 1

h 1[n]  
n
(

1)
 n
1
0.8
0.6
n0
0.4
0.2
0
2. Convolve the samples with h 1[n]
3. Reconstruct with a (t )
-0.2
-0.4
-0.6
-0.8
-1
-30
1.5
x(t)
The samples c
1
-10
0
10
20
30
0.5
0
0
-0.5
-0.5
-1
-1
-15
-10
-5
x
reconstructed x
1
0.5
-1.5
-20
-20
1.5
0
5
10
15
20
-1.5
-20
-15
-10
-5
0
5
10
15
20
17
Introduction
Input Signals
Nonlinearities
Reconstruction
Summary:
Perfect Recovery In A Subspace
General input signals (not necessarily BL)
General samples (anti-aliasing filters)
Results hold also for nonuniform sampling and more general spaces
18
Introduction
Nonlinearities
Input Signals
Reconstruction
Smoothness Prior
(Eldar 2007)
No subspace information but || Lx ||2  
Many consistent solutions S{xˆ}  c[n]
Motivation: Want xˆ (t ) to be close to x(t)
Minimize the worst-case difference:
min max || xˆ  x ||2 s.t. S{x}  c,|| Lx ||2  
xˆ
x
Complicated problem but … simple solution
c[n]
1
 S (  2 k )S * (  2 k )

S ( ) 
S ( )
| L( ) |2
xˆ (t )

  (t  n)
n 
optimal interpolation kernel
19
Introduction
Input Signals
Nonlinearities
Reconstruction
From Smoothness to Compressed Sensing
Non ideal sampling of analog signals
PR with subspace prior
Approximations with smoothness prior || Lx ||
Sparsity prior || Lx ||0: Discrete signals
Sparse prior:
x
Samples: c  a , x
2
cm1  Amn xn1 , m  n
Can x be reconstructed from c?
If every 2K columns of A are linearly independent then there is a
unique K-sparse signal (Donoho and Elad 03)
m  2K
Key observation: c can be relatively short and still contain the entire
information about x
20
Introduction
Input Signals
Nonlinearities
Reconstruction
Joint Sparsity
Multiple measurement vectors (MMV): C=AX
Each column of X is K-sparse
X 
The non-zero values share a common location set
Theorem
Let X have S  K . If
 ( A)  2 K  (rank(C )  1)
then X is the unique sparsest solution set
(Chen and Huo 06, Mishali and Eldar 07)
21
Introduction
Input Signals
Nonlinearities
Reconstruction
Algorithms
SMV
min || x || 0 s.t. c  Ax
Efficient algorithms:
Basis pursuit
Matching pursuit
Others
To overcome NP-hard
MMV
min | I ( X ) | 0 s.t. C  AX
Efficient algorithms:
M-Basis pursuit
M-Matching pursuit
Others
Results and Algorithms Inherently Discrete
22
Introduction
Input Signals
Nonlinearities
Reconstruction
Analog Compressed Sensing
What is analog compressed sensing?
A signal with a multiband structure in some basis
no more than N bands, max width B, bandlimited to
Previous methods for analog CS involve discretization or finite models
(R. Baraniuk , J. Laska, S. Kirolos, M. Duarte, T. Ragheb, Y. Massoud, A. Gilbert,
M. Iwen, M. Strauss, J. Tropp, M. Wakin, D. Baron)
Our model is inherently continuous: each band has an uncountable
number of non-zero elements
No finite basis!
23
Introduction
Input Signals
Nonlinearities
Reconstruction
Goals
1 Minimal rate
Blind
Sampling
Sampling
c[n]
2 Perfect
reconstruction
Blind
Reconstruction
Reconstruction
3 Blind system
24
Introduction
Input Signals
Nonlinearities
Reconstruction
Non-Blind Scenario
Theorem
Landau (1967)
Average sampling rate
is constant with Lebesgue measure
Minimal-rate sampling and reconstruction (NB) with known band
locations (Lin and Vaidyanathan 98) Subspace scenario
Half blind system (Herley and Wong 99, Venkataramani and Bresler 00)
25
Introduction
Input Signals
Nonlinearities
Reconstruction
Minimal Sampling Rate
Question:
What is the minimal sampling rate that allows blind perfect
reconstruction with arbitrary sampling/reconstruction methods?
Theorem
Mishali and Eldar (2007)
The minimal sampling rate is doubled
(for
)
Minimal sampling rate for our set M: 2NB
26
Introduction
Nonlinearities
Input Signals
Reconstruction
Sampling
Multi-Coset: Periodic Non-uniform on the Nyquist grid
In each block of
samples, only
are kept, as described by
2
3
Analog signal
Point-wise samples
0
0
3
2
0
2
3
27
Introduction
Input Signals
Nonlinearities
Reconstruction
The Sampler
DTFT
of sampling
sequences
Length .
known
Constant
matrix
known


in vector form

unknowns 


















28
Introduction
Input Signals
Nonlinearities
Reconstruction
Reconstruction Objectives
Goal:
Recover
Problems:
1. Undetermined system – non unique solution (p<L)
2. Continuous set of linear systems
Observation:
is sparse






















29
Introduction
Input Signals
Nonlinearities
Reconstruction
Uniqueness
Theorem
(Mishali and Eldar, 2007)

30
Introduction
Input Signals
Nonlinearities
Reconstruction
We Say NO to Discretization !
Choose a dense grid of
Solve
for each
Interpolate
Disadvantages:
Loose perfect reconstruction
Large computational complexity
Sensitivity to noise























31
Introduction
Input Signals
Nonlinearities
Reconstruction
Paradigm
Solve finite
problem
Reconstruct






















0
1
2
3
4
5
6
32
Introduction
Input Signals
Nonlinearities
Reconstruction
Once S is Known…
Solve finite
problem
Reconstruct
Reconstruct
exactly by
33
Introduction
Input Signals
Nonlinearities
Reconstruction
Continuous to Finite
CTF block
MMV
Solve finite
problem
Continuous
Reconstruct
Finite
34
Introduction
Input Signals
Nonlinearities
Reconstruction
CTF Fundamental Theorem
Theorem
Mishali and Eldar (2007)
35
Introduction
Input Signals
Nonlinearities
Reconstruction
Algorithm
CTF
Continuous-to-finite block: Compressed sensing for analog signals
Perfect reconstruction at minimal rate
Blind system: band locations are unkown
36
Introduction
Input Signals
Nonlinearities
Reconstruction
Summary: Perfect Reconstruction
Until Now:
Perfect reconstruction from subspace samples, sparse samples
Minimax reconstruction from smooth signals
Limitations:
Linear sampling
Ideal interpolation kernels
Coming up ….
Perfect compensation for nonlinear distortions
Minimax interpolation with simple kernels
37
Introduction
Input Signals
Nonlinearities
Reconstruction
Nonlinear Sampling
m(t )
x(t )
y (t )  A
c[n]
s(-t)
t=n
Memoryless nonlinear
distortion
Saturation in CCD sensors
Dynamic range correction
Optical devices
High power amplifiers
Many applications… No theory!
38
Introduction
Input Signals
Nonlinearities
Reconstruction
Perfect Reconstruction
Theorem (uniqueness):

If L2  A  S and m(t) is invertible and smooth enough
then y(t) can be recovered exactly
(Dvorkind, Eldar, Matusiak 2007)
Setting:
m(t) is invertible with bounded derivative
y(t) is lies in a subspace A
Uniqueness same as in linear case!
Proof: Based on extended frame perturbation theory and geometrical ideas
39
Introduction
Nonlinearities
Input Signals
Reconstruction
Algorithm: Linearization
Transform the problem into a series of linear problems:
1.
Initial guess y0
2.
Linearization: Replace m(t) by its derivative around y0
3.
Solve linear problem and update solution yn
c

yn  yn 1 m(t ) xn s(t )  error in
samples

PM '
yn
yn+1
( A), S

m 
'
1
yn
solving linear problem
correction
Questions:
1. Does the algorithm converge?
2. Does it converge to the true input?
40
Introduction
Nonlinearities
Input Signals
Reconstruction
Optimization Based Approach
Main idea:
1. Minimize error in samples ek
2. From uniqueness if ek  0
Perfect reconstruction
ek  S m( yk (t ))  c
 yk (t )  y(t )
where
global minimum of
ek
Difficulties:
1. Nonlinear, nonconvex problem
2. Defined over an infinite space
Theorem :
Under the previous conditions any stationary point of ek
is unique and globally optimal
(Dvorkind, Eldar, Matusiak 2007)
Our algorithm traps a stationary point!
41
Introduction
Input Signals
Nonlinearities
Reconstruction
Simulation Example
Optical sampling system:
y (t )  
optical modulator
x(t )
ADC
c[n]
42
Introduction
Input Signals
Nonlinearities
Reconstruction
Simulation
Third
iteration:
First
iteration:
Initialization
with y0  0
43
Introduction
Nonlinearities
Input Signals
Reconstruction
Constrained Reconstruction
s(t )
x(t )
H (e j )
c[n]
d (n)

xˆ (t )  W
w(t )
 (t  n)
reconstruction space
Consistent reconstruction (Unser and Aldroubi 94, Eldar 03,04)
xˆ (t )
s(t )
c[n]
Unique solution possible only if:
L2  W  S   xˆ  EWS  x
H (e j ) 
1

 W (  2 k )S
*
(  2 k )
k 
W
EWS  x
Problem: Resulting error can be quite large
x(t )
Ps x
S
44
Introduction
Nonlinearities
Input Signals
Reconstruction
Minimax Reconstruction
(Eldar and Dvorkind, 2005)
Motivation: Want xˆ (t ) to be close to x(t)
Best approximation in W is xˆ (t )  PW x(t ) but can’t be attained from c[ n]
Minimize the worst-case difference:
xˆ  PW x
max
min
xˆ
|| x|| L
2
Complicated problem but …. Simple solution: xˆ (t )  PW PS x(t )

 W (  2 k )S (  2 k )
*
H (e j ) 

k 
 | W (  2 k ) |  | S (  2 k ) |
2
k 
Comparison:
xˆ  x

2
k 
EWS  x
W
W
x(t )
Ps x
PW Ps x
S
Ps x
S
45
Introduction
Nonlinearities
Input Signals
Reconstruction
Example: Audio Processing
Down-up sampling with non-ideal filtering:
x(t )
8[kHz]
x(t )
LPF 1
H
/2
LPF 2
xˆ (t )
x2
Original signal
No processing (NE=0.81)
Consistent (NE=0.87)
Regret (NE=0.28)
Orthogonal Projection (NE=0.27)
46
Conclusion
Beyond Ideal Bandlimited Sampling
Broad class of input signals: Subspace priors, smoothness priors
Compressed sensing for analog signals
Compensations for many practical distortions
Applicable to a wide host of sampling problems
Can beat Nyquist and aliasing using the right tools!
47
References
Y. C. Eldar and T. Dvorkind, "A Minimum Squared-Error Framework for
Generalized Sampling," IEEE Trans. Signal Processing, vol. 54, no. 6, pp. 2155-2167,
June 2006.
M. Mishali and Y. C. Eldar, "Blind Multi-Band Signal Reconstruction: Compressed
Sensing for Analog Signals,“ submitted to IEEE Trans. on Signal Processing, Sep.
2007.
T. G. Dvorkind, Y. C. Eldar and E. Matusiak, "Nonlinear and Non-Ideal Sampling:
Theory and Methods," submitted to IEEE Trans. on Signal Processing, Nov. 2007.
M. Mishali and Y. C. Eldar, "Reduce and Boost: Recovering Arbitrary Sets of Jointly
Sparse Vectors", submitted to IEEE Trans. on Signal Processing, Feb. 2008.
Y. C. Eldar and M. Unser,"Nonideal Sampling and Interpolation from Noisy
Observations in Shift-Invariant Spaces," IEEE Trans. Signal Processing, Vol. 54, No. 7,
pp. 2636-2651, July 2006.
Y. C. Eldar, "Sampling and Reconstruction in Arbitrary Spaces and Oblique Dual
Frame Vectors ", J. Fourier Analys. Appl., vol. 1, no. 9, pp. 77-96, Jan. 2003.
O. Christensen and Y. C. Eldar, "Oblique Dual Frames and Shift-Invariant Spaces,"
Applied and Computational Harmonic Analysis, vol. 17/1, pp. 48-68, July 2004.
48
Details:
2
Ug 
2
1  sin( A, S )
2sin( A, S )
Lg 
0
1  sin( A, S )
x
m’(t)
Ug
Lg
y
where:
sin( A, S )  sup PS  f
f A
f 1
Maximal angle between the spaces
49