Transcript Generative Models of M/EEG: Group inversion and MEG+EEG+fMRI multimodal integration Rik Henson (with much input from Karl Friston)

Generative Models of M/EEG:
Group inversion and MEG+EEG+fMRI
multimodal integration
Rik Henson
(with much input from Karl Friston)
Overview
1. A Generative Model of M/EEG
2. Group inversion (optimising priors across subjects)
3. Multimodal integration:
3.1 Symmetric integration (fusion) of MEG + EEG
3.2 Asymmetric integration of MEG + fMRI
3.3 Full fusion of MEG/EEG + fMRI?
1. A PEB Framework for MEG/EEG
(Generative Model)
(Linear) Forward Model for MEG/EEG (for one timepoint):
Y = LJ + E
Y = Data
J = Sources
L = Leadfields
E = Error
n sensors
p>>n sources
n sensors x p sources
n sensors
(Gaussian) Likelihood:
p(Y | J)  N (LJ, C(e) )
C(e) = n x n Sensor (error) covariance
Prior:
p(J)  N (0, C( j ) )
C(j) = p x p Source (prior) covariance
Posterior:
p( J | Y )  p(Y | J) p(J)
Phillips et al (2005), Neuroimage
1. A PEB Framework for MEG/EEG
(Generative Model)
Specifying (co)variance components (priors/regularisation):
C   i Qi
i
C = Sensor/Source covariance p(X)  N (m, C)
Q = Covariance components
λ = Hyper-parameters
( e)
Empty-room:
# sensors
“IID” (white noise):
# sensors
1. Sensor components, Qi (error):
# sensors
# sensors
( j)
Multiple Sparse
Priors (MSP):
# sources
# sources
“IID” (min norm):
# sources
2. Source components, Qi (priors/regularisation):
# sources
Friston et al (2008) Neuroimage
1. A PEB Framework for MEG/EEG
(Generative Model)
( j)
Q1( j ) Q2
...
Q1(e) Q(2e)
i( e )
i( j )
C( e )
( j)
C
N (0, C)
N (0, C)
Ε
J
Fixed
Variable
...
L
Y
Data
Friston et al (2008) Neuroimage
1. A PEB Framework for MEG/EEG
(Inversion)
1. Obtain Restricted Maximum Likelihood (ReML) estimates of the
hyperparameters (λ) by maximising the variational “free energy” (F):
λˆ  max p(Y | λ )  max F


F  ln p ( Y | J , λ )  p ( J | λ )  ln q
q
2. Obtain Maximum A Posteriori (MAP) estimates of parameters (sources, J):
Jˆ  max p(J | Y, λˆ )  max F
j
j
ˆ ( j ) LT (LC
ˆ ( j ) LT + C
ˆ (e ) )-1 Y
C
cf. Tikhonov
…and an estimate of their posterior covariance (inverse precision):
ˆ ( j) - C
ˆ ( j ) LTC
ˆ -1LC
ˆ ( j)
ˆ C
Σ
ˆ  LC
ˆ ( j )LT  C
ˆ (e)
C
(relevant to MEG+EEG integration)
3. Maximal F approximates Bayesian (log) “model evidence” for a model, m:
ln p (Y | m)  ln  p(Y, J | m)dJ  F ( Jˆ , λˆ )
m  {L, Q, λˆ}
(relevant to MEG+fMRI integration)
Friston et al (2002) Neuroimage
1. A PEB Framework for MEG/EEG
Summary:
•
Automatically “regularises” in principled fashion…
•
…allows for multiple constraints (priors)…
•
…to the extent that multiple (100’s) of sparse priors possible…
•
…(or multiple error components or multiple fMRI priors)…
•
…furnishes estimates of source precisions and model evidence
2. Group Inversion
Specifying (co)variance components (priors/regularisation):
C   i Qi
i
C = Sensor/Source covariance p(X)  N (m, C)
Q = Covariance components
λ = Hyper-parameters
(e)
Empty-room:
# sensors
“IID” (white noise):
# sensors
1. Sensor components, Qi (error):
# sensors
# sensors
( j)
Multiple Sparse
Priors (MSP):
# sources
# sources
“IID” (min norm):
# sources
2. Source components, Qi (priors/regularisation):
# sources
Friston et al (2008) Neuroimage
2. Group Inversion
Specifying (co)variance components (priors/regularisation):
C   i Qi
i
C = Sensor/Source covariance p(X)  N (m, C)
Q = Covariance components
λ = Hyper-parameters
(e)
Empty-room:
# sensors
# sensors
“IID” (white noise):
# sensors
1. Sensor components, Qi (error):
# sensors
( j)
# sources
2. Optimise Multiple Sparse Priors by pooling across participants Qi
# sources
Litvak & Friston (2008) Neuroimage
2. Group Inversion (single subject)
(Generative Model)
( j)
Q1( j ) Q2
...
Q1(e) Q(2e)
...
i( e )
i( j )
C( e )
( j)
C
N (0, C)
N (0, C)
Ε
J
L
Y
Litvak & Friston (2008) Neuroimage
2. Group Inversion (multiple subjects)
(Generative Model)
( j)
Q1( j ) Q2
...
( e)
( e)
Q11
Q12
...
Q(21e)
Q(Ne1)
ik( e )
k( j )
C( e )
C( j )
N (0, C)
N (0, C)
Ε
J
Li
Y1
Y2
YN
Litvak & Friston (2008) Neuroimage
2. Group Inversion
(Generative Model)
~
 Y1   L1 
 E1  …projecting data and leadfields to a reference subject (0):
~   
E 
L
Y
 2    2 J   2 
~
T
T 1
Y
A

L
L
(
L
L
)
     
  
i  Ai Yi
i
0 i
i i
~   
 
L
Y
 N   N 
E N 
Common source-level priors:
C ( j )  (k j )Qk( j )
Subject-specific sensor-level priors:
Ci(e)  (ike) AiQk(e) AiT
C (e)
C1( e )

0


 

 0
0 

 
 0 
(e) 
0 C N 
0 
C2( e )

C  L0C ( j ) LT0  C (e)
Litvak & Friston (2008) Neuroimage
2. Group Inversion
(Generative Model)
MMN
MSP
MSP (Group)
Litvak & Friston (2008) Neuroimage
3. Types of Multimodal Integration
“Neural”
Activity
Causes (hidden):
(inversion)
Generative (Forward)
Models:
Data:
fMRI
Balloon
Model
Head Head
Model Model
MEG
?
EEG
? (future)
3. Types of Multimodal Integration
“Neural”
Activity
Causes (hidden):
Generative (Forward)
Models:
Data:
Balloon
Model
fMRI
Head Head
Model Model
MEG
?
EEG
Symmetric
Integration
(Fusion)
? (future)
Asymmetric
Integration
Daunizeau et al (2007), Neuroimage
3.1 Fusion of MEG+EEG
(Theory)
Specifying (co)variance components (priors/regularisation):
C   i Qi
i
C = Sensor/Source covariance p(X)  N (m, C)
Q = Covariance components
λ = Hyper-parameters
( e)
Empty-room:
# sensors
“IID” (white noise):
# sensors
1. Sensor components, Qi (error):
# sensors
# sensors
( j)
Multiple Sparse
Priors (MSP):
# sources
# sources
“IID” (min norm):
# sources
2. Source components, Qi (priors/regularisation):
# sources
Friston et al (2008) Neuroimage
3.1 Fusion of MEG+EEG
(Theory)
Specifying (co)variance components (priors/regularisation):
C   Q
(e)
i
( e)
ji
( e)
ij
j
Ci(e) = Sensor error covariance for ith modality
Qij = jth component for ith modality
λij = Hyper-parameters
( e)
Q21(e) 
# sensors
# sensors
E.g, white noise for 2 modalities:
Q11(e) 
# sensors
1. Sensor components, Qij (error):
# sensors
( j)
Multiple Sparse
Priors (MSP):
# sources
# sources
“IID” (min norm):
# sources
2. Source components, Qi (priors/regularisation):
# sources
Henson et al (2009) Neuroimage
3.1 Fusion of MEG+EEG
(Generative Model)
( j)
Q1( j ) Q2
Q1(e) Q(2e)
i( j )
i( e )
C( j )
C( e)
N (0, C)
N (0, C)
J
Ε
LMEG
YMEG
Henson et al (2009) Neuroimage
3.1 Fusion of MEG+EEG
(Generative Model)
( j)
Q1( j ) Q2
( e)
( e)
Q11
Q12
( e)
Q(21e) Q22
i( j )
ij( e )
C( j )
C1( e)
C(2e)
N (0, C)
N (0, C)
J
Ε
LMEG
LEEG
YMEG
YEEG
Henson et al (2009) Neuroimage
3.1 Fusion of MEG+EEG
(Theory)
• Stack data and leadfields for d modalities:
 Y1   L1 
 E1(1) 
   
 (1) 
Y2    L2  J   E2 
   


   
 (1) 
 Ed 
Yd   Ld 
C (e)
C1( e )

0


 

 0
0 

 
 0 

0 Cd( e ) 
0 
C2( e )

(note: common sources and source priors, but separate error components)
• Where data / leadfields scaled to have same average / predicted variance:
Yi 
Yi
1
mi
T
tr (YiYi )
Li 
Li
1
mi
T
i i
tr ( L L )
mi = Number of spatial modes
(e.g, channels)
Henson et al (2009) Neuroimage
3.1 Fusion of MEG+EEG
(Application)
ERs from 12 subjects for 3 simultaneously-acquired Neuromag sensor-types:
(Planar) Gradiometers
(MEG, 204)
Electrodes
(EEG, 70)
mV
fT
RMS fT/m
Magnetometers
(MEG, 102)
Faces
Scrambled
ms
ms
ms
Faces - Scrambled
150-190ms
Henson et al (2009) Neuroimage
3.1 Fusion of MEG+EEG
+31 -51 -15
MEG mags
MEG grads
+19 -48 -6
Faces
Scrambled
1 / ˆ ii  59
1 / ˆ ii  76
Faces – Scrambled,150-190ms
EEG
+43 -67 -11
1 / ˆ ii  95
IID noise for each modality; common MSP for sources
(fixed number of spatial+temporal modes)
FUSED
+44 -64 -4
1/ ˆ ii  127
Henson et al (2009) Neuroimage
3.1 Fusion of MEG+EEG
(Conclusions)
•
Fusing magnetometers, gradiometers and EEG
increased the conditional precision of the source
estimates relative to inverting any one modality alone
(when equating number of spatial+temporal modes)
•
The maximal sources recovered from fusion were a
plausible combination of the ventral temporal sources
recovered by MEG and the lateral temporal sources
recovered by EEG
•
(Simulations show the relative scaling of mags and
grads agrees with empty-room data)
Henson et al (2009) Neuroimage
3.2 Integration of M/EEG+fMRI
Specifying (co)variance components (priors/regularisation):
C   i Qi
i
C = Sensor/Source covariance p(X)  N (m, C)
Q = Covariance components
λ = Hyper-parameters
(e)
Empty-room:
# sensors
“IID” (white noise):
# sensors
1. Sensor components, Qi (error):
# sensors
# sensors
( j)
Multiple Sparse
Priors (MSP):
# sources
# sources
“IID” (min norm):
# sources
2. Source components, Qi (priors/regularisation):
# sources
Friston et al (2008) Neuroimage
3.2 Integration of M/EEG+fMRI
Specifying (co)variance components (priors/regularisation):
C   i Qi
i
C = Sensor/Source covariance p(X)  N (m, C)
Q = Covariance components
λ = Hyper-parameters
(e)
Empty-room:
# sensors
“IID” (white noise):
# sensors
1. Sensor components, Qi (error):
# sensors
# sensors
( j)
fMRI Priors:
# sources
# sources
“IID” (min norm):
# sources
2. Each suprathreshold fMRI cluster becomes a separate prior Qi
# sources
Henson et al (in press) Human Brain Mapping
3.2 Integration of M/EEG+fMRI
(Generative Model)
( j)
Q1( j ) Q2
...
Q1(e) Q(2e)
...
i( e )

( j)
i
C( e )
C( j )
N (0, C)
N (0, C)
Ε
J
L
Y
3.2 Integration of M/EEG+fMRI
(Generative Model)
YfMRI
Q1( j ) Q(2j )
Q3( j ) Q(4j )
Q1(e) Q(2e)
i( e )
i( j )
C( e )
( j)
C
N (0, C)
N (0, C)
Ε
J
L
YMEG
3.2 Integration of M/EEG+fMRI (Priors)
T1-weighted MRI
Anatomical data
{T,F,Z}-SPM
Functional data
…
1. Thresholding and connected component labelling
Gray matter
segmentation
Cortical surface
extraction
…
2. Projection onto the cortical surface
using the Voronoï diagram
…
3D geodesic
Voronoï diagram
3. Prior covariance components Qi( j )
Henson et al (in press) Human Brain Mapping
3.2 Integration of M/EEG+fMRI (Application)
1
2
4
5
SPM{F} for faces versus
scrambled faces,
15 voxels, p<.05 FWE
3
5 clusters from SPM of fMRI data from separate group of (18)
subjects in MNI space
Henson et al (in press) Human Brain Mapping
3.2 Fusion of MEG+fMRI (Application)
Negative Free Energy (a.u.)
(model evidence)
Magnetometers (MEG)
*
*
*
*
Gradiometers (MEG)
*
*
*
*
Electrodes (EEG)
*
None
*
*
Global
Local (Valid)
Local (Invalid)
Valid+Invalid
(binarised, variance priors)
Henson et al (in press) Human Brain Mapping
3.2 Fusion of MEG+fMRI (Application)
Negative Free Energy (a.u.)
(model evidence)
Magnetometers (MEG)
*
*
*
*
Gradiometers (MEG)
*
*
*
*
Electrodes (EEG)
*
None
*
*
Global
Local (Valid)
Local (Invalid)
Valid+Invalid
(binarised, variance priors)
Henson et al (in press) Human Brain Mapping
3.2 Fusion of MEG+fMRI (Application)
Negative Free Energy (a.u.)
(model evidence)
Magnetometers (MEG)
*
*
*
*
Gradiometers (MEG)
*
*
*
*
Electrodes (EEG)
*
None
*
*
Global
Local (Valid)
Local (Invalid)
Valid+Invalid
(binarised, variance priors)
Henson et al (in press) Human Brain Mapping
3.2 Fusion of MEG+fMRI (Application)
Negative Free Energy (a.u.)
(model evidence)
Magnetometers (MEG)
*
*
*
*
Gradiometers (MEG)
*
*
*
*
Electrodes (EEG)
*
None
*
*
Global
Local (Valid)
Local (Invalid)
Valid+Invalid
(binarised, variance priors)
Henson et al (in press) Human Brain Mapping
3.2 Fusion of MEG+fMRI (Application)
Negative Free Energy (a.u.)
(model evidence)
Magnetometers (MEG)
*
*
*
*
Gradiometers (MEG)
*
*
*
*
Electrodes (EEG)
*
None
*
*
Global
Local (Valid)
Local (Invalid)
Valid+Invalid
(binarised, variance priors)
Henson et al (in press) Human Brain Mapping
3.2 Fusion of MEG+fMRI (Application)
IID sources and IID noise (L2 MNM)
Magnetometers (MEG)
Gradiometers (MEG)
Electrodes (EEG)
None
Global
Local (Valid)
Local (Invalid)
Henson et al (in press) Human Brain Mapping
3.2 Fusion of MEG+fMRI (Application)
IID sources and IID noise (L2 MNM)
Magnetometers (MEG)
Gradiometers (MEG)
Electrodes (EEG)
None
Global
Local (Valid)
Local (Invalid)
Henson et al (in press) Human Brain Mapping
3.2 Fusion of MEG+fMRI (Application)
IID sources and IID noise (L2 MNM)
Magnetometers (MEG)
Gradiometers (MEG)
Electrodes (EEG)
None
Global
Local (Valid)
Local (Invalid)
fMRI priors counteract superficial bias of L2-norm
Henson et al (in press) Human Brain Mapping
3.2 Fusion of MEG+fMRI (Application)
IID sources and IID noise (L2 MNM)
Magnetometers (MEG)
Gradiometers (MEG)
Electrodes (EEG)
None
Global
Local (Valid)
Local (Invalid)
fMRI priors counteract superficial bias of L2-norm
Henson et al (in press) Human Brain Mapping
3.2 Fusion of MEG+fMRI (Application)
Differential Response
(Faces vs Scrambled)
Right Posterior Fusiform (rPF)
+26 -76 -11
Right Medial Fusiform (rMF)
Right Lateral Fusiform (rLF)
+32 -45 -12
+41 -43 -24
Gradiometers (MEG)
(5 Local Valid Priors)
L
Differential Response
(Faces vs Scrambled)
R
Differential Response
(Faces vs Scrambled)
Left occipital pole (lOP)
-27 -93 0
Left Lateral Fusiform (lLF)
-43 -47 -21
NB: Priors affect variance, not precise timecourse…
Henson et al (in press) Human Brain Mapping
3.2 Fusion of MEG+fMRI
(Conclusions)
• Adding a single, global fMRI prior increases model evidence
• Adding multiple valid priors increases model evidence further
Helpful if some fMRI regions produce no MEG/EEG signal
(or arise from neural activity at different times)
• Adding invalid priors rarely increases model evidence,
particularly in conjunction with valid priors
• Can counteract superficial bias of, e.g, minimum-norm
• Affects variance but not not precise timecourse
• (Adding fMRI priors to MSP has less effect)
Henson et al (in press) Human Brain Mapping
3.3 Fusion of fMRI and MEG/EEG?
“Neural”
Activity
Causes (hidden):
Fusion of fMRI +
MEG/EEG?
Data:
fMRI
Balloon
Model
Head Head
Model Model
MEG
?
EEG
? (future)
Henson (2010) Biomag
3.3 Fusion of fMRI and MEG/EEG?
( j)
Q1( j ) Q2
( e)
Q1(e) Q2
C( j )
C( e)
J
ΕMEG
s)
L(MEG
YMEG
Henson (2010) Biomag
3.3 Fusion of fMRI and MEG/EEG?
( j)
Q1( j ) Q2
( e)
Q1(e) Q2
(s)
A1(t ) Q2
C( j )
C( e)
ΕfMRI
J
ΕMEG
t)
H(fMRI
s)
L(MEG
YfMRI
YMEG
space (s)
time (t)?
Henson (2010) Biomag
Overall Conclusions
1. The PEB (in SPM8) framework is advantageous
2. Group optimisation of MSPs can be advantageous
3. Full fusion of MEG and EEG is advantageous
4. Using fMRI as (spatial) priors on MEG is advantageous
5. Unclear that fusion of fMRI and M/EEG is advantageous
The End
3. Fusion of MEG+EEG
Henson et al (2009) Neuroimage
log(λx106)
log(λx106)
Hyperparameters
3. Fusion of MEG+EEG
Henson et al (2009) Neuroimage
4. Fusion of MEG+fMRI
Gradiometers (MEG)
Electrodes (EEG)
Local
Valid
ln(λ)+32
fMRI hyperparameters
ln(λ)+32
Magnetometers (MEG)
Local
Invalid
Henson et al (in press) Human Brain Mapping