Transcript Recent Results on the Co-Sparse Analysis Model * Michael Elad The Computer Science Department The Technion – Israel Institute of technology Haifa 32000, Israel *Joint work.

Recent Results on the
Co-Sparse Analysis Model *
Michael Elad
The Computer Science Department
The Technion – Israel Institute of technology
Haifa 32000, Israel
*Joint work with
Contributed Session: Mathematical
Signal and Image Processing
The research leading to these results has
received funding from:
Ron Rubinstein Tomer Peleg
&
The European Council under the
European union's Seventh Framework
Programme (FP/2007-2013) ERC grant
Agreement ERC-SPARSE- 320649
Google Faculty Research
Award
Remi Gribonval, Sangnam Nam,
Mark Plumbley, Mike Davies,
Raja Giryes, Boaz Ophir,
Nancy Bertin
Informative Data  Inner Structure
Stock Market
Heart Signal
Still Image
Voice Signal
Radar Imaging
CT & MRI
 It does not matter what is the data you are working on – if it
carries information, it must have an inner structure.
Traffic info
 This structure = rules the data complies with.
 Signal/image processing relies on exploiting these “rules” by adopting models.
 A model = mathematical construction describing the properties of the signal.
 In the past decade, sparsity-based models has been drawing major attention.
The Co-Sparse Analysis Model:
Recent Results
By: Michael Elad
2
Sparsity-Based Models
Sparsity and Redundancy can be
Practiced in (at least) two different ways
Synthesis
The attention to
sparsity-based models
has been
given
mostly
This
Talk’s
to the synthesis
Message:option,
leaving the analysis
almost untouched.
The Co-Sparse Analysis Model:
Recent Results
By: Michael Elad
Analysis
a long-while
The For
co-sparse
analysis model is a very
these two options
Well … now we
appealing
alternative
to
the
synthesis
were confused,
know better !!
model;
it considered
has a great potentialThe
fortwo
signal
even
are
modeling;
BUT there are many
things
to be (near)VERY DIFFERENT
equivalent.
about
it we do not know yet
3
Agenda
Part I - Background
Recalling the Synthesis Sparse Model
Part II - Analysis
Turning to the Analysis Model
Part III – THR Performance
Revealing Important Dictionary Properties
Part IV – Dictionaries
Analysis Dictionary-Learning and Some Results
Part V – We Are Done
Summary and Conclusions
The Co-Sparse Analysis Model:
Recent Results
By: Michael Elad
4
Part I - Background
Recalling the
Synthesis Sparse Model
The Co-Sparse Analysis Model:
Recent Results
By: Michael Elad
5
The Sparsity-Based Synthesis Model
 We assume the existence of a synthesis
dictionary DIRdn whose columns are the
atom signals.
 Signals are modeled as sparse linear
combinations of the dictionary atoms:
D
…
x  D
 We seek a sparsity of , meaning that
it is assumed to contain mostly zeros.
 We typically assume that n>d: redundancy.
 This model is typically referred to as the
synthesis sparse and redundant
representation model for signals.
The Co-Sparse Analysis Model:
Recent Results
By: Michael Elad
x
=
D

6
The Synthesis Model – Basics
 The synthesis representation is expected
to be sparse:
 0  k  d
n
=
d
 Adopting a Bayesian point of view:
 Draw the support T (with k non-zeroes) at random;
Dictionary
D
 Choose the non-zero coefficients
randomly (e.g. iid Gaussians); and
α
x
 Multiply by D to get the synthesis signal.
 Such synthesis signals belong to a Union-of-Subspaces (UoS):
sp an D T 

x
w he re
DTT  x
T k
 This union contains
The Co-Sparse Analysis Model:
Recent Results
By: Michael Elad
n
 
k 
subspaces, each of dimension k.
7
The Synthesis Model – Pursuit
 Fundamental problem: Given the noisy measurements,
y  x  v  D  v,
v ~ N 0 ,  I
2
recover the clean signal x – This is a denoising task.
 This can be posed as: ˆ  A rgM in y  D 

2
2
s.t. 
0
 k  xˆ  D ˆ
 While this is a (NP-) hard problem, its approximated solution
can be obtained by
 Use L1 instead of L0 (Basis-Pursuit)
 Greedy methods (MP, OMP, LS-OMP)
 Hybrid methods (IHT, SP, CoSaMP).
Pursuit
Algorithms
 Theoretical studies provide various guarantees for the success of these
techniques, typically depending on k and properties of D.
The Co-Sparse Analysis Model:
Recent Results
By: Michael Elad
8
The Synthesis Model – Dictionary Learning
X
=
D

A

G iven Signals : y j  x j  v j v j ~ N 0 ,  I
M in D A  Y
D ,A
2
2
F
Example are
linear
combinations
of atoms from D
The Co-Sparse Analysis Model:
Recent Results
By: Michael Elad
s.t.  j  1 ,2,
N
j1
,N
j
0
k
Each example has a sparse
representation with no
more than k atoms
Field & Olshausen (`96)
Engan et. al. (`99)
…
Gribonval et. al. (`04)
Aharon et. al. (`04)
…
9
Part II - Analysis
Turning to the
Analysis Model
1.
2.
S. Nam, M.E. Davies, M. Elad, and R. Gribonval, "Co-sparse Analysis
Modeling - Uniqueness and Algorithms" , ICASSP, May, 2011.
S. Nam, M.E. Davies, M. Elad, and R. Gribonval, "The Co-sparse Analysis
Model and Algorithms" , ACHA, Vol. 34, No. 1, Pages 30-56, January 2013.
The Co-Sparse Analysis Model:
Recent Results
By: Michael Elad
10
The Analysis Model – Basics
* spark Ω
 The analysis representation z is expected to be sparse
Ωx
0
 z
0
 Co-Support:  - the rows that are orthogonal to x
Ω x  0
 This model puts an emphasis on the zeros in z for
characterizing the signal, just like zero-crossings of
wavelets used for defining a signal [Mallat (`91)].
p
 d1
d
p
 Co-sparsity: - the number of zeros in z.
T
=
Ω
x
z
Analysis Dictionary
 Co-Rank: Rank(Ω)≤ (strictly smaller if there are linear dependencies in Ω).
 If Ω is in general position*, then the co-rank and the co-sparsity are the same,
and 0   d , implying that we cannot expect to get a truly sparse analysis.
The Co-Sparse Analysis Model:
Recent Results
By: Michael Elad
11
The Analysis Model – Bayesian View
d
 Analysis signals, just like synthesis ones,
can be generated in a systematic way:
Synthesis Signals
Analysis Signals
Choose the
support T (|T|=k)
at random
Choose the cosupport  (||= )
at random
Coef. :
Choose T at
random
Choose a random
vector v
Generate:
Synthesize by:
DTT=x
Orhto v w.r.t. :
Support:
Ω
x
z
Analysis Dictionary
x   I  Ω  Ω   v
†
 Bottom line: an analysis signal x satisfies:  
The Co-Sparse Analysis Model:
Recent Results
By: Michael Elad
p
=
 
s.t. Ω  x  0 .
12
The Analysis Model – UoS
d
 Analysis signals, just like synthesis ones,
leads to a union of subspaces:
Synthesis
Signals
What is the Subspace
Dimension:
k
How Many Subspaces:
n
 
k 
Who are those Subspaces:
sp an D T 
Analysis
Signals
p
 
 

Ω
x
r=d-
span
p
=
z
Analysis Dictionary
Ω 
T

 The analysis and the synthesis models offer both a UoS construction, but
these are very different from each other in general.
The Co-Sparse Analysis Model:
Recent Results
By: Michael Elad
13
The Analysis Model – Count of Subspaces
 Example: p=n=2d:
 Synthesis: k=1 (one atom) – there are 2d subspaces of dimensionality 1.
 2d 


d1
>>O(2d) subspaces of dimensionality 1.
 In the general case, for d=40 and
p=n=80, this graph shows the
count of the number of subspaces.
As can be seen, the two models
are substantially different, the analysis
model is much richer in low-dim.,
and the two complete each other.
 The analysis model tends to lead to
a richer UoS. Are these good news?
10
10
10
10
15
10
Synthesis
Analysis
5
Sub-Space dimension
10
The Co-Sparse Analysis Model:
Recent Results
By: Michael Elad
20
# of Sub-Spaces
 Analysis: =d-1 leads to
0
0
10
20
30
40
14
The Low-Spark  Case
 What if spark(T)<<d ?
 For example: a TV-like operator for imagepatches of size 66 pixels ( size is 7236).
 Here are analysis-signals generated for cosparsity ( ) of 32:
Ω D IF
 H o rizo ntal 


D e rivative






V e rtical


 D e rivative 
800
700
 Their true co-sparsity is higher – see graph:
 In such a case we may consider  d , and thus
… the number of subspaces is smaller.
# of signals
600
500
400
300
200
100
0
0
10
20
30
40
50
60
70
80
Co-Sparsity
The Co-Sparse Analysis Model:
Recent Results
By: Michael Elad
15
The Analysis Model – Pursuit
 Fundamental problem: Given the noisy measurements,
y  x  v,
 
s.t . Ω  x  0 ,
v ~ N 0 ,  I
2
recover the clean signal x – This is a denoising task.
 This goal can be posed as:
xˆ  A rgM in y  x
x ,
2
2
s.t. Ω  x  0
&  p
o r rank  Ω    d  r
 This is a (NP-) hard problem, just as in the synthesis case.
 We can approximate its solution by L1 replacing L0 (BP-analysis), Greedy
methods (BG, OBG, GAP), and Hybrid methods (AIHT, ASP, ACoSaMP, …).
 Theoretical study providing pursuit guarantees depending on the co-sparsity
and properties of . See [Candès, Eldar, Needell, & Randall (`10)], [Nam, Davies, Elad, &
Gribonval, (`11)], [Vaiter, Peyré, Dossal, & Fadili, (`11)], [Peleg & Elad (’12)].
The Co-Sparse Analysis Model:
Recent Results
By: Michael Elad
16
The Analysis Model – Backward Greedy
BG finds one row at a time from  for approximating the solution of
xˆ  ArgM in y  x
x ,
i  0 , xˆ 0  y  0  
2
2
s.t. Ω  x  0 & Rank  Ω    d  r 
Stop condition?
(e.g. Rank  Ω    d  r )

Yes
Output xi
No
i  i  1 ,  i   i  1  A rgM in w k xˆ i  1
T
k  i  1
The Co-Sparse Analysis Model:
Recent Results
By: Michael Elad
†
xˆ i   I  Ω  i Ω  i  y
17
The Analysis Model – Backward Greedy
Is there a similarity to a
synthesis pursuit algorithm?
Synthesis
OMP
Other options:
Stop[Rubinstein,
condition?
Peleg &Yes
Elad (`12)] Output x=
i  0 , xˆr•00 Optimized
y  0   BG
 pursuit (OBG)
Rank
(e.g. Rank
 Ω D   d k r )
• Greedy Analysis Pursuit (GAP) [Nam,
Davies, Elad & Gribonval (`11)]
No [Giryes, Nam, Gribonval & Davies (`11)]
• Iterative Cosparse Projection
• Lp relaxation using IRLS [Rubinstein (`12)]
• CoSAMP/SP like algorithms
[Giryes, et. al. (`12)]
T
†

ˆ




A
rgM
in
w
x
i  i  1, i
M ax dk r ii11
rxˆii   I  D
Ω D
Ω   y
i1


• Analysis-THR [Peleg
k  & Elad (`12)]
i1
The Co-Sparse Analysis Model:
Recent Results
By: Michael Elad
i
y-ri
i
18
Synthesis vs. Analysis – Summary
m
 The two align for p=n=d : non-redundant.
 Just as the synthesis, we should work on:
D
d
 Pursuit algorithms (of all kinds) – Design.
=
α
 Pursuit algorithms (of all kinds) – Theoretical study.
 Dictionary learning from example-signals.
 Applications …
d
 Our work on the analysis model so far touched
on all the above. In this talk we shall focus on:
 A theoretical study of the simplest pursuit method:
Analysis-THR.
 Developing a K-SVD like dictionary learning method
for the analysis model.
The Co-Sparse Analysis Model:
Recent Results
By: Michael Elad
x
p
=
Ω
x
z
19
Part III – THR Performance
Revealing Important
Dictionary Properties
1.
T. Peleg and M. Elad, Performance Guarantees of the Thresholding Algorithm for the
Co-Sparse Analysis Model, IEEE Transactions on Information Theory, Vol. 59, No. 3,
Pages 1832-1845, March 2013.
The Co-Sparse Analysis Model:
Recent Results
By: Michael Elad
20
The Analysis-THR Algorithm
Analysis-THR aims to find an approximation for the problem
xˆ  A rgM in y  x
x ,
C o m pute z  Ω y & so rt
2
2
s.t.  Ω  x  0 & R ank  Ω    d  r 
i  0,  0  

(incre asing)    k k  1
p
Stop condition?


R ank Ω  i  d  r
No
Yes
Output
†
xˆ   I  Ω  i Ω  i  y
No
i  i  1 ,  i   i 1   i
The Co-Sparse Analysis Model:
Recent Results
By: Michael Elad
21
The Restricted Ortho. Projection Property
r 
m in
 ,j
R ank  Ω    d  r
and j
I  Ω  Ω   w j
wj
Ω
The Co-Sparse Analysis Model:
Recent Results
By: Michael Elad
2
†
m ax
 ,j
Ω Ω  w j
R ank  Ω    d  r
and j
2
 ROPP aims to get near
orthogonality of rows
outside the co-support
(i.e., αr should be as close
as possible to 1).
 This should remind of the
(synthesis) ERC [Tropp (’04)]:

T
1
†
Ω
m ax
S ,j S  k & j S
†
DSdj
1
1
22
Theoretical Study of the THR Algorithm
r
Choose 
Such that
Choose
Ω
p d
Generate
e
R ank  Ω    d  r

 
Ω
 0 ,  I 
2
Project
x
†
u

 0 ,I  

Co-Rank
r
y
xˆ
The Analysis
THR
Algorithm
ˆ
u
d
P r suce ss i.e. ˆ  
Ω
e
x  I  Ω  Ω   u
Generate
d

The Co-Sparse Analysis Model:
Recent Results
By: Michael Elad


 m ax   1 
0
 
   
e xp 

2

8


8
2
p   dr

 
 2Q 
 r




p




23
Implications
Prob(Success)
Theoretical
Results
Prob(Success)
As empirical tests show,
the
theoretical performance
predicts an improvement
Co-Sparsity
for an Ω with strong

linear dependencies,
Prob(Success)
Noise Power
and high ROPP
Empirical
Results
r
ROPP
The Co-Sparse Analysis Model:
Recent Results
By: Michael Elad
24
Part IV – Dictionaries
Analysis Dictionary-Learning
and Some Results
1.
2.
B. Ophir, M. Elad, N. Bertin and M.D. Plumbley, "Sequential Minimal Eigenvalues - An
Approach to Analysis Dictionary Learning", EUSIPCO, August 2011.
R. Rubinstein T. Peleg, and M. Elad, "Analysis K-SVD: A Dictionary-Learning Algorithm
for the Analysis Sparse Model", IEEE-TSP, Vol. 61, No. 3, Pages 661-677, March 2013.
The Co-Sparse Analysis Model:
Recent Results
By: Michael Elad
25
Analysis Dictionary Learning – The Signals
X
Ω
=
Z
We are given a set of N contaminated (noisy)
analysis signals, and our goal is to recover their
analysis dictionary, 
y
j
 xj  vj,   j 
The Co-Sparse Analysis Model:
Recent Results
By: Michael Elad

s.t . Ω  j x j  0 , v ~ N 0 ,  I
2
N
j1
26
Analysis Dictionary Learning – Goal
Synthesis
M in D A  Y
D ,A
2
F
s.t.  j  1 ,2,
,N
j
,N
Ωxj
0
k
Analysis
M in X  Y
Ω ,X
2
F
Noisy Examples
s.t.  j  1 ,2,
0
p
Denoised Signals are L0 Sparse
We shall adopt a similar approach to the K-SVD for
approximating the minimization of the analysis goal
The Co-Sparse Analysis Model:
Recent Results
By: Michael Elad
27
Analysis K-SVD – Outline [Rubinstein, Peleg & Elad (`12)]
Ω
X
…
=
.
Initialize Ω
The Co-Sparse Analysis Model:
Recent Results
By: Michael Elad
Sparse Code
Z
…
.
Dictionary
Update
28
Analysis K-SVD – Sparse-Coding Stage
Ω
X
=
…
Z
.
.
M in X  Y
Ω ,X
2
F
s.t.  j  1 ,2,
,N
Ωx j
0
p
Assuming that  is fixed, we
aim at updating X

xˆ j  ArgM in x  y j
X
The Co-Sparse Analysis Model:
Recent Results
By: Michael Elad
2

N
2
s.t. Ω x
…
0
p
These are N separate
analysis-pursuit
problems. We suggest
to use the BG or the
OBG algorithms.
j1
29
Analysis K-SVD – Dictionary Update Stage
Ω
X
…
=
Z
.
.
M in X  Y
Ω ,X
2
F
s.t.  j  1 ,2,
,N
…
Ωx j
0
p
• Only signals orthogonal to the atom should get to vote for its
new value.
• The known supports should be preserved.
• Improved results for applications are obtained by promoting
linear dependencies within Ω.
The Co-Sparse Analysis Model:
Recent Results
By: Michael Elad
30
Analysis Dictionary Learning – Results (1)
Experiment #1: Piece-Wise Constant Image
Initial 
 We take 10,000 6×6 patches (+noise σ=5) to train on
 Here is what we got
(we promote sparse
outcome):
Trained
(100 iterations)

Original Image
Patches used for training
The Co-Sparse Analysis Model:
Recent Results
By: Michael Elad
31
Analysis Dictionary Learning – Results (2)
Experiment #2: denoising of the piece-wise constant image.
256256
Non-flat patch examples
The Co-Sparse Analysis Model:
Recent Results
By: Michael Elad
32
Analysis Dictionary Learning – Results (2)
Learned dictionaries for =5
Analysis Dictionary
38 atoms
(again, promoting
sparsity in Ω)
The Co-Sparse Analysis Model:
Recent Results
By: Michael Elad
Synthesis Dictionary
100 atoms
33
Analysis Dictionary Learning – Results (2)
Synthesis
K-SVD
BM3D
n
Average subspace
dimension
d Patch denoising:
error per element
d
Image PSNR [dB]
n/a
n/a d
2.42
2.03
1.75
1.74
1.79
1.69
1.51
1.43
2.91
5.37
1.97
4.38
7.57
10.29
6.81
9.62
40.66
35.44
43.68
38.13
46.02
39.13
32.23
30.32
34.83
32.02
35.03
31.97
Cell Legend:
The Co-Sparse Analysis Model:
Recent Results
By: Michael Elad
Sparse Analysis
K-SVD
=5
=10
=15
=20
34
Analysis Dictionary Learning – Results (3)
Experiment #3: denoising of natural images (with =5)
The following results
were obtained by
modifying the DL
algorithm to improve
the ROPP
Barbara
House
Method
Barbara
House
Peppers
Fields of Experts (2009)
37.19 dB
38.23 dB
27.63 dB
Synthesis K-SVD
38.08 dB
39.37 dB
37.78 dB
Analysis K-SVD
37.75 dB
39.15 dB
37.89 dB
Peppers
An Open Problem: How to “Inject” linear
dependencies into the learned dictionary?
The Co-Sparse Analysis Model:
Recent Results
By: Michael Elad
35
Part V – We Are Done
Summary and
Conclusions
The Co-Sparse Analysis Model:
Recent Results
By: Michael Elad
36
Today …
Sparsity and
Redundancy are
practiced mostly in
the context of the
synthesis model
• The differences between
the two models,
• A theoretical study of
the THR algorithm, &
• Dictionary learning for
the analysis model.
Today we
discussed
Is there any
other way?
Yes, the analysis model is
a very appealing (and
different) alternative,
worth looking at
In the past few years
there is a growing
interest in this model,
deriving pursuit
methods, analyzing
them, designing
dictionary-learning, etc.
So, what
to do?
These slides and the relevant papers can be found in
http://www.cs.technion.ac.il/~elad
37
Thank you for your time,
and …
Thanks to the organizers:
Gitta Kutyniok and Otmar Scherzer
Questions?
The Co-Sparse Analysis Model:
Recent Results
By: Michael Elad
38
The Analysis Model – The Signature
Consider two possible dictionaries:
Ω D IF
R an d o m Ω
1
0.8
0.6
Random 
DIF
Relative
number of
linear
dependencies
0.4
0.2
# of rows
0
0
Sp ark  Ω
T

4
Sp ark  Ω
The Co-Sparse Analysis Model:
Recent Results
By: Michael Elad
T
  37
10
20
30
40
The Signature of a matrix is more
informative than the Spark.
Is it enough?
39