Transcript Document
Propagating beliefs in spinglass models
Yoshiyuki Kabashima
Dept. of Compt. Intel. & Syst. Sci.
Tokyo Institute of Technology
Hayashibara@SMAPIP
@ 15/7/2003
Tokyo Institute of Technology
1
Background and Motivation
Active research on belief propagation (BP) in
information sciences (IS)
Similarity to methods in physics
TMM & Bethe approx.
Difference in interest
Physics ⇒ obtained solutions
IS ⇒ dynamics of the algorithm
may cause unexpected developments in both
fields
Hayashibara&SMAPIP
@15/7/2003
Tokyo Institute of Technology
2
Purpose and Results
Purpose:analyze dynamics of BP when
employed in spin-glass models
Results:
Macro. dyn. of BP ⇒ RS solution
Micro. stability of BP ⇔ AT condition
Hayashibara&SMAPIP
@15/7/2003
Tokyo Institute of Technology
3
Outline
SG model in Bayesian framework
Belief propagation
Macro. dyn. and RS solution
Micro. stability and AT condition
Summary
Hayashibara&SMAPIP
@15/7/2003
Tokyo Institute of Technology
4
Spin-glass models
SG models on a random (Bethe) lattice
M
H S | J J
1
S
l
lL
K-body interaction
C-bonds/spin
Randomly constructed for other aspects
1 J0 / C J
1 J0 / C J
PJ
J J / C
J J / C
2
2
Hayashibara&SMAPIP
@15/7/2003
Tokyo Institute of Technology
5
SM and Bayesian Statistics
Boltzmann dist. = Bayes formula
PJ | S
M
exp H S | J
Z J
1
PJ | S
M
S
exp J Sl
lL
PJ | S
2 cosh J
PS | J
1
Magnetization = Posterior average
ml
Hayashibara&SMAPIP
@15/7/2003
S
S
l
exp H S | J
Z J
Sl PS | J
S
Tokyo Institute of Technology
6
Graph Expression
Expression by a bipartite graph
K 3
J1
J3
J2
J4
JM
…
PJ1 | S
…
…
S1
Hayashibara&SMAPIP
@15/7/2003
S2
S3
C 4 PS2 | J
Tokyo Institute of Technology
SN
7
Belief Propagation
Iterative inference by passing beliefs
J1
J3
J2
J4
JM
…
…
t
ml
ˆ l
m
t
…
S1
Hayashibara&SMAPIP
@15/7/2003
S2
S3
Tokyo Institute of Technology
SN
8
More Precisely
mˆ t l 1 tanh J mt k
kL \ l
t
1
t
ml tanh tanh mˆ l
M l \
t
ml :Posterior average when J is left out.
ˆ t l
tanh 1 m
: Effective field when
J
comes in.
:Estimator
mlt tanh tanh 1 mˆ t l
M l
Hayashibara&SMAPIP
@15/7/2003
Tokyo Institute of Technology
9
Macro. Dyn. vs. RS Solution
Distribution (histogram) of beliefs
x m , ˆ xˆ (1 / NC ) xˆ mˆ t l
l 1 M l
Known result for finite C:
N
x (1 / NC)
t
l 1
N
t
M l
t
l
Tree approximation (resampling graph/update)
Density evolution ⇒ RS solution
K 1
K 1
t 1
t
ˆ
ˆ
ˆ
x
dx
x
x
tanh
J
x
l
l
l
l 1
l 1
J
C 1
C 1
t
1
t x
dx ˆ xˆ x tanh tanh xˆ
1
1
Richardson & Urbanke (2001) Vicente, Saad, YK (2000)
Hayashibara&SMAPIP
@15/7/2003
Tokyo Institute of Technology
10
Novel Result for Infinite C
Central limit theorem for infinite C
Evolution of average and variance
h Et
exp
t
C 1
C 1
2
F
t h dxˆ ˆ t xˆ h tanh 1 xˆ
1
1
2F t
2
Natural iteration of RS SP eqs.
2
t 1
t K 1
t 1
t K 1
, F J Q
E J 0 M
t
t
t
2
M
Dz
tanh
F
z
E
,
Q
Dz
tanh
Hayashibara&SMAPIP
@15/7/2003
Tokyo Institute of Technology
F z E
t
11
Experimental Validation
SK model(N=1000,J=1,T=0.5)
J 0 1.5
(AT stable)
Hayashibara&SMAPIP
@15/7/2003
J 0 0.5
1
t
D
N
(AT unstable)
m
N
l 1
Tokyo Institute of Technology
t
l
t 1 2
l
m
12
Microscopic Instability
Possible microscopic instability while BP
seems to macroscopically converge
Stability analysis of the fixed point
ht l 1
M l \
tanh J
m
kL
k
\l
1 tanh J mk
kL \ l
Hayashibara&SMAPIP
@15/7/2003
2
jL
Tokyo Institute of Technology
\l
1 m2j
mj
htj
13
Evolution of Perturbation
Dist. of Perturb. f t y 1 / NC y ht l
l 1 M l
Perturbation Evolution
N
f
t 1
f y y is attractive
⇔ the fixed point(=RS solution) is stable
C 1, K 1
y dyl f t yl
1,l 1
Hayashibara&SMAPIP
@15/7/2003
K 1
tanh J xl
2
C
1
K
1
1 xk
l 1
y
y k
2
K 1
1
k 1 xk
1
tanh
J
x
l
l 1
Tokyo Institute of Technology
J , xl
14
Pictorial Expression
What is performed?
(t 1) th update
tth update :
J1
J3
J2
S1
S2
t
h21
J4
S3
t
h23
JM
…
…
…S
J1
J1
J3
J2
S1
S2
J4
S3
J4
JM
…
…
…
SN
S1
N
t 1
h21
t
h24
Hayashibara&SMAPIP
@15/7/2003
J3
J2
Tokyo Institute of Technology
S2
S3
JM
…
…
…S
N
t 1
t 1
h23
h24
15
Meaning of P. Evolution
Link to known results for infinite C
Central limit theorem:
t
1
y
a
f t y
exp
t
t
2
b
2b
2
:Gaussian dist.
P. Evolution → Update of average & variance
t 1
K 2
t
a
(
K
1
)
J
M
1
Q
a
0
t 1
2
K 2
2
b
(
K
1
)
J
Q
Dz
1
tanh
Hayashibara&SMAPIP
@15/7/2003
Tokyo Institute of Technology
2
Fz E b O a
t
t 2
16
Meaning of P. Evolution
Critical conditions for growth of
fluctuation
K 2
1 Q 1 :Average
(
K
1
)
J
M
0
2
K 2
2
(
K
1
)
J
Q
Dz
1
tanh
FzE
1 :Variance
2
For K=2 (SK model)
Average → Tf: Para-Ferro transition
Variance → TAT: AT condition
Hayashibara&SMAPIP
@15/7/2003
Tokyo Institute of Technology
17
Analysis for finite C
Is P. evolution equivalent to AT analysis even
for finite C?
AT analysis for finite C is complicated.
But, P. evolution is (numerically) possible.
K=2 (Wong-Sherrington model)
Paramagnetic solution
Known result
Average→T
C
1
tanh
J
1
f
J
•Klein et al (1979)
2
C 1 tanh J 1 Variance→TAT •Mezard & Parisi
J
(1987)
Hayashibara&SMAPIP
@15/7/2003
Tokyo Institute of Technology
18
Analysis for Finite C
Ferromagnetic solution
Numerical evaluation of Tpevol: New result!
N=2000, K=2, C=4,
20000MCS/Spin
D: Tpevol in Ferro phase
Hayashibara&SMAPIP
@15/7/2003
Tokyo Institute of Technology
19
Summary
Close relationship between BP and the replica
analysis
Macro. dyn. ⇒ RS solution
Micro. stability ⇔AT condition
This correspondence may be useful for AT
analysis for SG models of finite connectivity.
Application to CDMA multiuser detection
(Kabashima, to appear in J. Phys. A, 2003)
Hayashibara&SMAPIP
@15/7/2003
Tokyo Institute of Technology
20