Transcript Diapositiva 1 - Hippocampal Protocol

XV PMT meeting – Jan 30, 2013
Agenda:
- Hippocampal Label Expansion project
- Certification Platform: performance/criteria
- Questionnaire
- Congress presentations
- Papers
4
Label Expansion - Design
Subjects = 135
Variable
MTA
Value range
0-4
Hippocampi = 270
Levels
N Subjects
x level
N Hippos
x level
5
27
54
Siemens, GE,
Philips
Magnet field 1.5 T, 3T
strength
Diagnosis
CTRL, MCI, AD
3
~ 45
90
2
66 +
132 +
3
~ 45
~ 90
Age
60-70, 70-80, 80+
3
~ 45
~ 90
Sides
Right, Left
2
135
135
Scanner
Hippo Label Expansion
USD 95239
AA (50%), Bioclinica, Brain Image Analysis, Ixico, Roche, Synarc, True
Positive Medical Devices (Louis Collins, Jan 29) (50%)
Laval:
OK managing labels for qualification of Algorithms
Fund from Can AA will better enable that, and completion of current
platform/certification criteria for human tracers (Wide logistical analysis
for certification criteria can be carried out by Abderazzak at Laval)
Deadline: 3 months from $ receipt
Tracers:
1) Masami (Jaccard=83 in the whole Certification phase) 60 images
(overlap with his beta tester project for Jap VSRAD algorithm)
2) Martina 19 images
Contacted: Rossana (Master; 28 img?), Dominik (Master; 28 img?)
Publication policy
AVAILIABILITY OF DELIVERABLES:
Needs being changed?
NO for the qualification labels (they should not be delivered anyway)
Maybe for the labels from the validation. Anyway, SD notes that
these labels are FEW, and from few subjects:
developers will have all what they need from the Expansion project,
 no need to change policy on availability of labels
Publication policy
AVAILIABILITY OF DELIVERABLES
The Harmonized Protocol (i.e. the text description and illustrative
material), the digital master tracer masks, and digital training sets
(masks developed by the 5 best naïve tracers) will be available only
to beta-test users in the period (about 1 year) between the end of
the Delphi Panel and the publication of the full validation resultsOr
ADFEQUATE QC (i.e. the analysis of the whole set of hippocampal
tracings by all naïve tracers).
To qualify as beta-testers, users will submit proposals to the SC; if
accepted, beta-testers will be granted access to deliverables under
conditions dictated by a written cooperation agreement that will be
signed by the user and the Chair of the SC. Assuming mutual
benefit, the cooperation agreement will include provisions related to
scientific publications, donations, and feedback that beta users will
provide on their exploitation of the Harmonized Protocol.
ADNI Publication policy
AVAILIABILITY OF LABELS ON THE “SODIUM” WEBSITE
This would soon be an issue for the Expansion
Expanded labels may be back-transformed to native space:
users will only need the ADNI code
Platform: Training Phase
2 Images
(Scheltens=0,3)
Feedback
6 Images (Scheltens=0,1,2,3,4,4)
TRAINING
Feedback
CERTIFICATION
10 Images (2 x Scheltens=0,1,2,3,4)
Feedback
VALIDATION
Training Performance & Validation
Tracer
I round
II round
III round
QUALIFICATION
Valid phase 1
Valid phase 2
BL
By End of June?
1
Kristian
Complete
Complete
Complete
Complete
2
Corinna
Complete
Complete
Complete
Complete
3
Mariangela
Complete
Complete
Complete
Complete
4
Tim Swih.
Complete
Complete
Complete
Complete
5
Masami
Complete
Complete
Complete
Complete
6
Michel
Complete
Complete
Complete
Complete
End March
7
Travis
Complete
Complete
Complete
Complete
Feb 6
8
Marileen
Complete
Complete
Complete
Complete
31 genn
9
Enrica
Complete
Complete
Complete
Complete
31 genn
10
Adam
Complete
Complete
1 Febbraio
11
Chadwich
Complete
Complete
15 feb
12
Margo
Complete
Complete
13 genn
13
Yawu
Complete
Complete
Complete
Complete
Feb 6
14
Melanie
Complete
Complete
10 genn
15
Emma
Complete
Feb 11
16
Oliver
Complete
31 jan
17
Gregory
QUALIFIED
QUALIFIED
QUALIFIED
END
June 15
31 genn
fase 2? (100 img)
Feb 6
END
Jan 31
ON
fase 2? (100 img)
fase 2? (100 img)
fase 2? (100 img)
Ros
Proposed max 30 sbj
Dominik
Proposed max 28 sbj
Martina
Max 19 sbj
Emma, round 1 (1.5T), Jaccard: 0.69 left and 0.65 right
Yawu, round 1 (1.5T), tot Jaccard: 0.71
does not distinguish amy from hippo; includes fornix
Melanie, img: bq_f_15T, total Jaccard: 0.76
She does not separate properly the indusium griseum, and sometimes includes
fornix. These errors are repeated in more slices where the criterion applies. All the
rest is ok or really minor.
Masami, img: bq_a_3T, total Jaccard: 0.82
Minor mistake in applying the HP criteria, or small inconsistencies in potentially
ambiguous boundaries
These are all of his divergence points. We can not really say that he is “not
compliant” with the HP
Michel,
subject
bq_F_3T
(Qualific)
Jaccard:
left
0.873,
right
0.870
Jaccard indices in Qualification
Michel
0,85
Masami
0,83
Marileen
0,81
Tim
0,81
Kristian
0,80
Corinna
0,80
Yawu
0,80
Travis
0,79
Enrica
Mariangela
0,79
0,78
Reliability in III round
Overlapping indices versus Benchmark segmentations
Inter-rater ICCs among
the 10 “Naïve” tracers
(95% CI)
3 Tesla
Dice
1.5 Tesla
Jaccard
Dice
3 Tesla + 1.5 Tesla
Jaccard
Right
left
N=10 MRIs – 6 corrected segmentations from previous training rounds included
Median
0.91
0.83
0.90
0.81
0.98
0.98
Minimum
0.89
0.80
0.82
0.74
(0.95-0.99)
(0.95-0.99)
Maximum
0.92
0.85
0.91
0.84
N=4 new MRIs - corrected segmentations from previous training rounds excluded
Median
0.90
0.82
0.90
0.82
0.89
0.91
Minimum
0.88
0.78
0.71
0.63
(0.69-0.99)
(0.72-0.99)
Maximum
0.92
0.86
0.92
0.86
Certification Criteria (to be defined)
Hypothesis so far (generated versus quality check) is that:
Jaccard < 70 denotes major problems in hippocampal segmentation
Jaccard < 80 denotes incomplete compliance with the HP criteria for
normal subjects (may be admissible for atrophic subjects: Enrica,
bq_r_15T, J=0.724, tracings ok)
Combination of values needed
(Wide statistical analysis needed. See slide 3, Abderazzak at Laval
and Can AA funds for Laval)
Questionnaire
How long does it take to you to segment one hippo based on the
HP?
Do you believe that your comprehension of the hippocampus and
your way to segment has improved after the training based on HP?
Which are the parts of the HP that were less clear to you?
Were there regions in the Hippocampus where you did not know
how to segment due to lack of instruction for that specific point or
case? (if yes, please attach figures illustrating them)
Questions regarding the platform as required by SD
We can use SD’s Survey Monkey’s account
Congress Presentations
- Biomarkers for Brain Disorders (Cambridge, February 3-5, 2013)
Definition of harmonized protocol for hippocampal segmentation
- AD/PD (Florence, March 6-10, 2013)
Definition of Harmonized Protocol for Hippocampal Segmentation
- AAN (San Diego, March 16-23, 2013)
EADC-ADNI Benchmark Images of Harmonized Hippocampal Segmentation
- IEEE Medical Measurement and Applications conference (Gatineau
–Quebec-, May 4-5, 2013.
Platform for Training and Qualification
- AIC-AAIC (Boston, July 13-18, 2013)
Training of Naïve Tracers; Statistics for Certification Platform
-(SINDEM (Ita Soc for Neurology of Dementias) Perugia March 13-15
Definition of harmonized protocol for hippocampal segmentation
Papers describing the project
DONE
Survey of protocols (preliminary phase; Published, JAD 2011)
Operationalization (preliminary phase; Accepted, Alzheimer’s &
Dementia, MS n. ADJ-D-12-00094)
Axes check short report (Brescia Team, ADJ-D-13-00030)
IN PROGRESS
Delphi consensus (Brescia Team, in progress)
Master tracers’ practice and reliability (Brescia Team, in progr)
Development of certification platform (Duchesne and coll)
PLANNED
Validation data (Brescia Team – companion paper 1)
Protocol definition (Brescia Team – companion paper 2)
Validation vs pathology (TBD)
VALIDATION VS CURRENT PROTOCOLS
ASSESSMENT OF SOURCES OF VARIANCE
TRAINING SET DEVELOPMENT
GOLD STANDARD
VALIDATION VS PATHOLOGY
20 naïve tracers
5 master tracers
1 tracer
Local Protocol
ADNI scans: 2 x 5 Scheltens’s atrophy scores
x 2 sides x 2 magnet strengths (1.5-3T)
QUALIFICATION
Harmonized Protocol
ADNI scans: 2 x 5 Scheltens’s atrophy score x
2 sides x 2 magnet strength (1.5-3T)
Total per rater: 40 hippos
Assessment of variance due to rater and center
Training
ADNI scans: 10 at 1.5T x 2 sides x 7 SUs
x 2 tracing rounds
Total per rater: 40 hippos
Harmonized Protocol
ADNI scans: 2 x 5 Scheltens’s atrophy
score x 2 sides
x 2 magnet strengths (1.5-3T)
Total per rater: 40 hippos
REFERENCE PROBABILISTIC MASKS
with 95% C.I.
QUALIFICATION
Harmonized Protocol:
Pathological datasets:
Mayo Clinic and NYU
Total: about 40 hippos
Best 5 naïve tracers
Harmonized Protocol
ADNI scans: 2 sides x 5 Scheltens’s atrophy scores x 3
time points (bl-1y-2y) x 3 scanners (+ retracing @ bl)
x 2 magnet strengths (1.5-3T)
Total per rater: 240 hippos
Assessment of variance due to side, trace-retrace,
atrophy, time, scanner, rater
TRAINING SET
Assessment of agreement with
volume on pathology or ex vivo MRI
and correlation with neuronal density
GANTT