Transcript Interalgorithm Study using synthetic nodules

Slide - 1
Confidential
Slide - 2
Confidential
Interalgorithm Study using CT
Images of synthetic
nodules……….
Slide - 3
Confidential
Objectives of the QIBA 3A Group

An inter-algorithm study, in the same way QIBA has been working on inter-reader,inter-scanner, and inter-site. We will also connect
it to the analysis section of QIBA Profile.
aim of the study is to estimate inter- and intra-algorithm variability
 The
by the volume estimation of synthetic nodules from CT scans of an
anthropomorphic phantom (according to the work of the QIBA 1A
Group (see Dr. Petrick‘s paper , SPIE 2011)
 For the impelmentation of the obejctives a challenge could be organized (?)
Slide - 4
Confidential
Study Motivation
Motivation for the Study
Study Design
Dataset
Algorithms
Analysis Protocol
Analysis method
Result evaluation
Result presentation
Result evaluation for the QIBA protocol
Slide - 5
Confidential
Motivation
Slide - 6
Confidential
Motivation






Changes in nodules volume is important for diagnosis, therapy planning, therapy response evaluation
Measuring volume changes requires high accuracy in measurement of absolute volume
Ground true has to be exact measured. This is not the case by data annotation (inter- and intra-observer
variability)
Volumes of synthetic nodules are measured (high accuracy)
Therefore it make sense to use such data (as ground truth) in order to calculate accuracy measurement of
algorithms
The study results could be combined with the QIBA 1A Group work. This combination will improve the
QIBA volumetric CT Profile development.
study has to be complementary to the study of the 1A
 The
Group:
(inter- intra 1A study: Radiologists and synthetic nodulesobserver
variability)
intra-algorithms
 3A Study: Algorithms and synthetic nodules:variability)

Using of the same technological basis (scanner type, Philips (1A Group data, synthetic nodules)) to secure
independence for inter-scanner variability (except BIAS between different serial numbers)
Slide - 7
Confidential
Analysis procedure (analysis Protocol according Bio-change or
Volcano Challenge?)

Describe the overall analysis procedure (use as example the Bio-change Challenge Protocol
http://www.nist.gov/itl/iad/dmg/biochangechallenge.cfm: How to Participate in Biochange
Challenge (Dr Fenimore)
Download and read the Bio-change Challenge Protocol and email or fax Statement of Interest to NIST.
Download the Bio-changeChallengeSeries from the NIST FTP site and from the NBIA RIDER collection as
described in the Protocol.
Run your change analysis algorithm or CAD tool in your lab on the validation data.
Report your change results in one of the required formats and send a Participation Agreement signed by the
your team leader to NIST by January 18, 2011.
NIST will analyze the reported results, comparing them to the limited available ground truth as described in the
Protocol. NIST will provide Participants with individual analysis of their results. We will publish the results of
the evaluation, without publicly identifying individual scores by Participant.
Slide - 8
Confidential
Has to be discussed during the meeting March 17th 2011
OUR CONTEXT is QIBA
 The aim of the study is not a challenge to know:
 Who is the best image analysis algorithm
 The aim of the study is to gain knowledge for the
QIBA profile (paragraph 9 concerns image analysis)
 Avoid competition
Support Cooperation, conjoint approach
Slide - 9
Confidential
Data/Nodules/Algorithms


Phantom data as used for the 1A Group Study (see Dr. Patrick’s publication, SPIE 2011)
1C clinical data (Dr Fenimore/ Phantom –synthetic data)



Training dataset
Test dataset
Or only test dataset (analyze without trainings dataset)
data pool specification (simple geometries, up to anatomical representations)
(if clear than we could propose an experimental design)

Nodule classification concerning position. Shape and margins of the nodule … (according to the QIBA Profile)

Algorithm classification

Analysis protocol:
Slide - 10




each algorithm is applied to the dataset separately
each algorithms is applied to the “training” data set
each algorithms is applied to the “test” data set
Statistical analysis of the analysis results of each algorithm separately
Confidential
Algorithm vendors and Algorithm classification examples from the
literature


Academia and non profit organizations
Industrial vendors
(for example possible vendors, according to the Volcano 2009 challenge could be: Siemens, Phipps, MeVis, Kitware, Definiens, VIA CAD etc…

Description/Classification of the algorithms: according to the grade of user intervention is needed (for example Volcano’09, A. P. Reeves et al):







Totally automatic using seed points
Limited parameter adjustment (on less than 15% of the cases)
Moderate parameter adjustment (on less than 50% of the cases)
Extensive parameter adjustment (more than 50% of the cases)
Limited image/boundary modification (on less than 15% of the cases)
Moderate image/boundary modification (on less than 50% of the cases)
Extensive image/boundary modification (more than 15% of the cases)
Or:

M. Gavrielide et al., Noncalcified Lung Nodules, Volumetric assessment with thoracic CT (RSNA 2009):


Semi automated
Manually derived
Semi automated algorithms are typically initiated by defining a region of interest around a nodule or by a user-provided point inside
the nodule area. Depending on the application, segmentation algorithms are then employed to delineate nodules from the surrounding lung
parenchyma and neighboring structures such as attached vasculature and pleural surfaces.
Manually derived methods require users to interactively delineate nodule boundaries, typically in a section-by-section fashion; this is
followed by an application of three-dimensional software to merge the two-dimensional boundaries into a volume. The estimate of nodule
volume is then based on the total number of voxels within the segmented region.
The majority of volume measurement methods use voxel counting
Slide - 11
Confidential
IEEE Transactions on medical imaging, vol. 28 MICCAI
Algorithm Classification according 3D Segmentation in the Clinic:
A Grand Challenge II - Liver Tumor Segmentation, Xiang Deng and Guangwei Du, Center for Medical Imaging Validation, Siemens Corporate Technology
(citation)
Abstract: In this paper, we present the organization of a competition of liver tumor segmentation techniques. The liver tumor segmentation competition is part of
the workshop "3D Segmentation in the Clinic:
A Grand Challenge II" at Medical Image Computing and Computer Assisted Intervention 2008 conference. The goal of this contest is to compare the he
performance of different algorithms for segmenting liver tumor from contrast enhanced CT images. Several organizing topics are described, contrast including the
motivation for organizing this competition, training and testing dataset evaluation methods
Algorithm Classification:
 An automatic segmentation algorithm does not require any user intervention.
 A semi- automatic algorithm needs minimal amount of input from user, e.g., a
seed point to initialize the segmentation.
 An interactive algorithm requires manual editing of the final results.
Slide - 12
Confidential
MICCAI Liver Challenge proposed metrics (citation)
Slide - 13
Confidential
Slide - 14
Confidential
Lucila Ohno-Machado
An introduction to calibration and discrimination methods, HST951 Medical Decision Support
Harvard Medical School, Massachusetts Institute of Technology

Sensitivity
Sens = TP/TP+FN

Specificity
Spec = TN/TN+FP

Slide - 15
Accuracy = (TN +TP)/all
Confidential
Other
 Box plot (for example: for each data set, nodule volume vs. method,
relative Volume)
Volrel= 100%* (Volest – Voltrue)/ Voltrue
Volrel= 100%* (Volalg – Volsynthetic nodule)/ Volsynthetic nodule
We propose: A simple but well defined and reliable approach
Slide - 16
Confidential
We propose: keep it simple
Algorithm Classification:
•An automatic segmentation algorithm does not require any user
intervention.
•A semi- automatic algorithm needs minimal amount of input from
user, e.g., a seed
point to initialize the segmentation.
•An interactive algorithm requires manual editing of the final results.
Slide - 17
Confidential
Algorithm Evaluation Method
Known: Vsn
Measurement: Valg
StdvValg (1) <- same Phantom, multiple measurements, same algorithm, multiple
seed points (two groups of algorithms: semiautomatic, manually)
StdvValg (2)<- different phantoms (according QIBA profile: nodule classification), same algorithm:
• StdValg (nodule class)
• StdValg(all nodule classes)
Bias out of digitalization in the area of lesion margins (Voxels/Pixels on the lesion margins, mixed
pixels) (Group 1A, Group 1C, Group 1B, QIBA VIA Profile, Definitions and how should be calculated? )
Achievements:
•
Transparent and Objective comparability between different algorithms
•
One independent Error value for further investigations of clinical data
Slide - 18
Confidential
Definition of the Study Data/Measurements/Metrics /
 Which data:

1A (Phantom/synthetic data) (defined experimental conditions, experimental design
Phantom Data for dV?)

1B Clinical data
according to that: do we have we to separate the study design as
following?
 Study design for Phantom/Synthetic data (focus)
 Study design for Clinical data

Slide - 19
Metrics
Confidential
Decision
Slide - 20
Confidential
Result information for the participants
 Each participant has to be informed (only their own algorithm results)
 Publication of the results (all participants)
 Using the results for the QIBA protocol: knowledge exploitation for the
QIBA protocol
 Resulting recommendations for FDA
Slide - 21
Confidential
(Is this possible?)