Transcript September 2012 Update September 13, 2012 WITH FUNDING SUPPORT PROVIDED BY NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY Andrew J. Buckler, MS Principal Investigator, QI-Bench.

September 2012 Update
September 13, 2012
WITH FUNDING
SUPPORT
PROVIDED BY
NATIONAL
INSTITUTE OF
STANDARDS
AND
TECHNOLOGY
Andrew J. Buckler, MS
Principal Investigator,
QI-Bench
Agenda for Today
• Update on statistical analysis library modules,
including conceptual development of aggregate
uncertainty (Jovanna)
• Overview of functionality in Reference Data Set
Manager staged for the development iteration
(Patrick)
2
Unifying Goal of 2nd Development
Iteration
• Perform end-to-end characterization of vCT including
meta-analysis of literature, incorporation of QIBA
results, and "scaled up" using automated detection and
reference volumetry method.
• Integrated characterization across QIBA, FDA,
LIDC/RIDER, Give-a-scan, Open Science sets (e.g.,
biopsy cases), through analysis modules and rolling up
to an i_ file in zip archive.
• Specifically have people like Jovanna, Ganesh, and
Adele to use it (as opposed to only Gary, Mike/Patrick,
and Kjell)
3
Analyze: Update on Library Modules
Module
Specification
Repository
Status
Meta-analysis
extraction
Extract observation data from literature
reports
CalculateReadingsFromStatistics.R
CalculateReadingsAnalytically.m
CalculateReadingsFromMeanStdev.m
Three reporting styles are supported
Method
Comparison
Radar plots and related methodology
based on readings from multiple
methods on data set with ground truth
CMI3A_1 5 PercentErr
v1_13_sent11May2012.R
Currently have 3A pilot in R, not yet
generalized but straightforward to do so.
Plan to refine based on Metrology
Workshop results and include case of
comparison without truth also.
Bias and
Linearity
According to Metrology Workshop
specifications
AnalyzeBiasAndLinearity.R
Complete except for minor code
improvements.
Variability and
Variance
Components
Analysis
Accepts as input fractional factorial
data of cross-sectional biomarker
estimates with range of fixed and
random factors, produces mixed effects
model
PerformBlandAltmanAndCCC.R
ModelLinearMixedEffects.R
Complete at this stage of development,
but presently the support for multiple
timepoints does not use repeated
measures analysis which it probably
should.
Treatment
Effect and
Variance
Components
Assessment
Accepts as input longitudinal change
data, estimates variance due to
treatment and non-treatment factors
ModelLinearMixedEffectsCHG.R
Prototyped, presently being refined.
Aggregate
Uncertainty
Accepts as input multiple s_ files,
produce intermediate i_ file result, and
then use it to derive aggregate based
on components
CalculateAggregateUncertainty.m
Prototyped (with team discussion today)
4
Analyze: Validation
• Go to Jovanna’s desktop
5
Analyze: Aggregate Uncertainty
• Objective: comprehensively characterize the performance of an
imaging biomarker.
• Two orthogonal considerations:
– Breadth of data used: use as much data as you can, regardless of where it
comes from!
– Nature of study designs that result in determination of uncertainty
components
• Approach:
– Utilize common analytical pipeline to place literature and heterogeneous
study results onto a common plane (this motivates the file conventions
that drive the library design)
– Roll-up separate components into an aggregate: current WIP for
discussion
6
Analyze: Aggregate Uncertainty
• Go back to Jovanna’s desktop
7
Execute: Basic Plan
• Generalize processing framework from the previous
development year
• Support user-in-the-loop processing workflows for
certain data-processing tasks
• Refine input and output formats to adhere to newer
standards (AIM 4.0, DICOM Segmentation Objects, etc.)
8
Execute: Implementation
• Support for Radiological Worklists and DICOM
Query and Retrieve directly from Execute
– Batchmake scripts initiate worklist item delegation
and reader stations can retrieve those datasets from
Midas as they would a PACS
• Generalize processing API harness to allow
arbitrary algorithm runs on arbitrary datasets.
• Optimize the web API and scripting interface to
allow more seamless interaction with other QIBench applications
9
10
Value proposition of QI-Bench
• Efficiently collect and exploit evidence establishing
standards for optimized quantitative imaging:
– Users want confidence in the read-outs
– Pharma wants to use them as endpoints
– Device/SW companies want to market products that produce them
without huge costs
– Public wants to trust the decisions that they contribute to
• By providing a verification framework to develop
precompetitive specifications and support test
harnesses to curate and utilize reference data
• Doing so as an accessible and open resource facilitates
collaboration among diverse stakeholders
11
Summary:
QI-Bench Contributions
• We make it practical to increase the magnitude of data for increased
statistical significance.
• We provide practical means to grapple with massive data sets.
• We address the problem of efficient use of resources to assess limits of
generalizability.
• We make formal specification accessible to diverse groups of experts that are
not skilled or interested in knowledge engineering.
• We map both medical as well as technical domain expertise into
representations well suited to emerging capabilities of the semantic web.
• We enable a mechanism to assess compliance with standards or
requirements within specific contexts for use.
• We take a “toolbox” approach to statistical analysis.
• We provide the capability in a manner which is accessible to varying levels of
collaborative models, from individual companies or institutions to larger
consortia or public-private partnerships to fully open public access.
12
QI-Bench
Structure / Acknowledgements
•
Prime: BBMSC (Andrew Buckler, Gary Wernsing, Mike Sperling, Matt Ouellette, Kjell Johnson, Jovanna
Danagoulian)
•
Co-Investigators
–
–
•
•
Financial support as well as technical content: NIST (Mary Brady, Alden Dima, John Lu)
Collaborators / Colleagues / Idea Contributors
–
–
–
–
–
–
•
Georgetown (Baris Suzek)
FDA (Nick Petrick, Marios Gavrielides)
UMD (Eliot Siegel, Joe Chen, Ganesh Saiprasad, Yelena Yesha)
Northwestern (Pat Mongkolwat)
UCLA (Grace Kim)
VUmc (Otto Hoekstra)
Industry
–
–
•
Kitware (Rick Avila, Patrick Reynolds, Julien Jomier, Mike Grauer)
Stanford (David Paik)
Pharma: Novartis (Stefan Baumann), Merck (Richard Baumgartner)
Device/Software: Definiens, Median, Intio, GE, Siemens, Mevis, Claron Technologies, …
Coordinating Programs
–
–
RSNA QIBA (e.g., Dan Sullivan, Binsheng Zhao)
Under consideration: CTMM TraIT (Andre Dekker, Jeroen Belien)
13