Transcript RCT Schema The Trial Bank Project Ida Sim, MD, PhD Associate Professor of Medicine Director, Center for Clinical and Translational Informatics University of California, San.

RCT Schema
The Trial Bank Project
Ida Sim, MD, PhD
Associate Professor of Medicine
Director, Center for Clinical and Translational Informatics
University of California, San Francisco, CA
Supported by
The Trial Bank Project R01-LM-06780
National Center for Biomedical Ontology U54 HG004028-01
Copyright 2007, Ida Sim
Outline
• Background
• RCT Schema
»
»
»
modeling approach
the class structure
evaluation
• Relationship to CTO
• Summary
Copyright 2007, Ida Sim
Goals of the CTO
• (1) fully and faithfully capture the types of entities and
relationships involved in clinical trials
• (2) comprehend terms like: cohort, randomization, placebo, etc.,
including ... statistical terms and terms for ... meta-analysis;
• (3) organize these terms in a structured way, providing definitions
and logical relations designed to enhance retrieval of, reasoning
with, and integration of the data annotated in its terms
• ...
• (6) draw on and seek maximal alignment with existing clinical
trial ontologies, including:
»
RCT Schema ontology used by theTrial Bank Project
Copyright 2007, Ida Sim
Major Axes for Aligning Models
• Domain of clinical trials
»
»
»
design: (non-)randomized, crossover, cluster randomized,
factorial...
objective: interventional, diagnostic, preventive...
clinical domain: drugs, procedures, organizational change...
• Task
»
»
trial design, execution, reporting, analysis, application
for individual trials, sets of clinically related trials
• Purpose (application vs. domain ontology)
»
»
to support accomplishment of domain task(s)
to define shared meaning for “integration of the data annotated
in its terms”
Copyright 2007, Ida Sim
Trial Tasks
Trial Ap plication
Guidelines
Trial Interpretation
Electronic
Patient Record
Decision M odels
Systematic Rev iew
Tri a l Bank
Trial Registration
Trial Design
Scientific Reporting
Fe edback to
Trial De s ign
Trial Conduct
Regulatory Reporting
Trial Execution
Copyright 2007, Ida Sim
Trial Bank Definition
• Computable repository of RCT information sufficiently
detailed to support scientific analysis for
»
»
designing future clinical trials
evidence-based practice and policy making
• Detailed information on
»
»
»
study design
study execution
summary and individual participant-level results
• Trial Bank is NOT for running a trial
Copyright 2007, Ida Sim
Trial Bank Target Uses
Trial Ap plication
Guidelines
Trial Interpretation
Electronic
Patient Record
Decision M odels
Systematic Rev iew
Tri a l Bank
Trial Registration
Trial Design
Scientific Reporting
Fe edback to
Trial De s ign
Trial Conduct
Regulatory Reporting
Trial Execution
Copyright 2007, Ida Sim
Trial Bank Software
• RCT Bank built on RCT Schema
»
Ocelot frame-based ontology
• Bank-a-Trial
»
»
web-based program for trialists to enter trial instances into RCT Bank
clinical descriptions of trial features (slot values) are in UMLS
• RCT Presenter
»
web-based browser of individual trials
Bank-a-Trial
RCT Presenter
RCT Bank
RCT Schema
Copyright 2007, Ida Sim
Major Axes for RCT Schema
• Domain of clinical trials
»
»
»
design: (non-)randomized, crossover, cluster randomized,
factorial...
objective: interventional, diagnostic, preventive...
clinical domain: drugs, procedures, organizational change...
• Task
»
»
trial design, execution, reporting, analysis, application
for individual trials, sets of clinically related trials
• Purpose
»
»
to support accomplishment of domain task(s) [application ontology]
to define shared meaning for “integration of the data annotated in
its terms”
Copyright 2007, Ida Sim
Outline
• Background
• RCT Schema
»
»
»
modeling approach
the class structure
evaluation
• Relationship to CTO
• Summary
Copyright 2007, Ida Sim
Entity Specification Problem
• What RCT aspects to model in RCT Schema? What
not to model?
»
»
multiple users (e.g., trialists, systematic reviewers)
multiple tasks (e.g., analysis, interpretation)
– multiple methods
»
no one correct RCT ontology
• Need principled, systematic approach
»
to specifying, documenting, and evaluating
Copyright 2007, Ida Sim
Competency Decomposition
Method
(Sim, et al, JBI 2004: 37(2):108-119)
• To define the entities that must be in a conceptual
model
• General approach
»
»
»
specify a task hierarchy of target tasks and subtasks
specify methods for each task
specify entities required for completing each task
using each method
• Generates a specification of required entities
»
the information requirements for the competencies
(tasks and subtasks) that the model/knowledge base
is to support
Copyright 2007, Ida Sim
Target Task for RCT Bank
• Using RCTs for trial design or
clinical application requires
synthesizing evidence across
all trials on a topic
Trial Interpretation
Trial Ap plication
Guidelines
Electronic
Patient Record
Decision M odels
Systematic Review
RCT Bank
Trial Registration
Trial Design
Scientific Reporting
Feedba ck to
Tri a l Design
Trial Conduct
Regulatory Reporting
Trial Execution
Copyright 2007, Ida Sim
Systematic Reviewing
• Canonical method for synthesizing evidence across
trials
• Major steps are
»
»
»
retrieve related RCTs (e.g., 32 trials of metformin for
diabetes)
analyze how comparable the trials are
statistically combine data if appropriate
– combining smaller trials increases statistical power to detect
effects
Copyright 2007, Ida Sim
Target Task = Systematic Review
• RCT Bank entities = sys. review information needs
»
identified all systematic review tasks
– review of literature and personal experience conducting 3
systematic reviews
»
»
»
identified methods for completing these tasks
organized tasks and methods into a task hierarchy
derived RCT entities necessary and sufficient for each
(sub)task
Copyright 2007, Ida Sim
Top-Level Sys Review Tasks
• Trial retrieval
• Trial critiquing
»
»
judging internal validity
judging generalizability
• Meta-analysis of quantitative results
»
analysis of clinical and statistical heterogeneity
• Contextual interpretation
»
scientific, socio-economic, and ethical
http://rctbank.ucsf.edu/tasks/tasks.html
Copyright 2007, Ida Sim
Judgment of Generalizability
• Were the people enrolled in the trial representative and
unbiased?
»
»
were eligible patients randomly selected from the source
population?
were enrolled subjects a random subset of those
eligible?
• Are the trial subjects similar to mine?
• Do I have the tested intervention available here?
Copyright 2007, Ida Sim
Method-(In)dependent Entity
Specification
• Were the people enrolled in the trial representative
and unbiased?
»
were eligible patients randomly selected from the
source population?
– method: no computable algorithm available
 recruitment method
»
were enrolled patients a random subset of those
eligible?
– method: using standard statistics
 number and clinical characteristics of enrolled subjects
 number and clinical characteristics of eligible but nonenrolled subjects
Copyright 2007, Ida Sim
Results: Entity Requirements
High-level
Tasks
Retrieval
SubTask I
Methods
SubTask II
(n=35)
SubTask III
(n=74)
Critiquing
Internal
Validity
1
1
Contextual
Interpretation
.....
External
Validity
3
...
2
2
Quantitative
Synthesis
4
1
...
...
...
...
11
4
7
2
...
...
...
39
13
22
29
30
171 Unique Entities Required 112
...
9
Copyright 2007, Ida Sim
Entity Specification Evaluation
• Evaluated match between
»
»
the 171 information items
388 requirements in 18 published trial-critiquing
instruments
• Results
»
»
entity specification is comprehensive
entity specification is reasonable
Sim, et al, KR-MED 2004; JBI 2004: 37(2):108-119
Copyright 2007, Ida Sim
Benefits of Approach
• Task hierarchy understandable by domain experts
»
identifies which information items are required for which
tasks
• Provides an evaluation “yardstick”
»
»
if an ontology contains all the information requirements
for a task
– then is it “competent” for that task
can evaluate and compare application ontologies
• Documents an (application/domain) ontology
»
»
states which tasks an ontology is competent for, and why
cross-indexes tasks and entities in the ontology
Copyright 2007, Ida Sim
Outline
• Background
• RCT Schema
»
»
»
modeling approach
the class structure
evaluation
• Relationship to CTO
• Summary
Copyright 2007, Ida Sim
Implementation of 171 Items
• Purpose of RCT Bank KB is to support scientific
analysis of trial evidence
»
needed a “data-schema” or “instance-style” ontology
Bank-a-Trial
RCT Presenter
RCT Bank
RCT Schema
• RCT Schema implemented the 171 information
items in a frame-based ontology
Copyright 2007, Ida Sim
RCT Schema Ontology
• Ocelot frame-based model
»
7 levels deep
• 192 frames, 607 unique slots
»
»
»
avg. 9.8 slots/frame
3 frames (1.6%) have multiple parents
193/607 slots (32%) take other frames as values
• Available at http://rctbank.ucsf.edu/
Copyright 2007, Ida Sim
• RCT Schema displayed in
GKB Editor
»
»
classes are red boxes
instances are blue boxes
• Class hierarchy organizes
entities as
»
»
trial concepts
trial descriptions (details)
• Not fully compliant with
“Werner’s Rules”
Includes IS-A Hierarchies
Copyright 2007, Ida Sim
Instantiating RCT Schema
• Clinical content described by terms from a clinical
vocabulary, e.g.,
»
for a breast cancer trial, instance of BASELINECHARACTERISTIC is described by
– term “menopause” from UMLS preferred term
– the UMLS CUI
»
Trial Bank software supports any vocabulary in
UMLS (e.g., SNOMED)
• Each trial is a collection of instances of classes
Copyright 2007, Ida Sim
• 518-TRIAL
»
»
»
518BACKGROUNDDETAILS
518-ADMINDETAILS
518-EXECUTEDPROTOCOL
– 518-ALL-SUBJECTS
– 518-PRIMARYOUTCOME-1
– etc.
»
»
518-ERRATUM
518-CONCLUSIONDETAILS
• 518-PRIMARY-OUTCOME-1 (e.g., all-cause mortality)
»
518-STAT-ANALYSIS-AND-RESULTS-1 (e.g. t-test)
– 518-ALL-COMPARISONS-AT-TIME-X-1 (e.g., at 6
months)
 518-SINGLE-TIME-X-COMPARISON-1 (e.g., between
PCI and thrombolysis groups)
• 518-SINGLE-TIME-X-COMPARISON-1
»
datapoint for PCI group
– numerator (all-cause deaths at 6 months)
– denominator (had all-cause death outcome assessed at 6 months)
– 518-STUDY-ARM-POPULATION-1 (the PCI group)
»
»
»
datapoint for thrombolysis group
summary odds ratio under intention-to-treat analysis
summary odds ratio under efficacy analysis
Outline
• Background
• RCT Schema
»
»
»
modeling approach
the class structure
evaluation
• Relationship to CTO
• Summary
Copyright 2007, Ida Sim
Expressivity Evaluation
Characteristic
Clinical domains
Intervention
Types
Examples
Cardiology, Radiology, Geriatrics, etc.
Procedures (thrombolysis), Single and Multistep Drugs (aspirin, warfarin), Counseling,
Multiple interventions in one arm
Outcome Types Dichotomous, continuous, univariate,
multivariate, survival, regression, scored
instruments (e.g., Wechsel Memory Scale)
Result Types
Intention-to-treat, efficacy analysis, subgroup
analyses
• Captured 17 full and 20+ partial trials
Copyright 2007, Ida Sim
Modeling Challenges Met
• Multi-armed, crossover, cluster randomized studies
• Many variations of patient drop-out, loss to followup
(e.g., excluded after randomization)
• For each outcome, the # of subjects assessed at each
timepoint in each subgroup
• Blinding efficacy: did subjects know which arm they
were assigned to?
• etc.
Copyright 2007, Ida Sim
Modular, Extensible
• Extensions possible with only minimal changes to 192
existing classes
New
Classes
New
Slots
Old Classes
Changed
Cluster randomized trials
2
10
3
Complex intervention regimens
4
8
3
Computable eligibilty rules
?
?
3
Modeling Extension
• Extensible to new clinical domains (e.g., genomics) via
clinical vocabularies
»
no clinical terms in RCT Schema
Copyright 2007, Ida Sim
Limitations of Representation
• Modeled, not yet tested
»
»
»
participant-level data
factorial designs
designs with run-in and washout periods
• In development
»
computable eligibility rules
• Not yet modeled
»
»
»
genomic data
nested subgroups
secondary studies (e.g,. followup studies)
Copyright 2007, Ida Sim
Trial Bank Publishing
• How to get trials into RCT Bank?
• Collaborated with JAMA and Ann Int Med to explore copublishing trials as articles and RCT Bank entries
»
»
»
authors submit manuscripts for peer review as usual
trial-bank staff enter accepted trials into trial bank
co-published 14 trials in RCT Presenter
• Evaluation
»
83 respondents evaluated a trial using both RCT
Presenter and the Journal Article
– mostly trialists and meta-analysts
Copyright 2007, Ida Sim
RCT Presenter Evaluation
100
90
80
70
60
50
40
30
20
10
0
12
12
85
81
65
85
Same as Article
•
12
23
27
31
65
39
50
39
46
42
50
73
42
70% of respondents
rated RCT Presenter
as good as or better
than the Journal
Article for all
attributes
M
Cl
or
ea
e
re
or
Ea
M
r
ga
or
si
e
er
ni
t
ze
r
fo
us
d
r
tw
ap
M
or
pr
M
or
th
ai
or
e
y
si
e
un
n
us
g
de
tr
ef
r
ia
ul
st
l
a
fo
n
da
r
cl
bl
in
e
W
ic
a
ou
lc
ld
ar
us
e
e
m
W
or
ou
e
ld
pr
ef
er
si
er
Ea
st
er
15
Fa
% Respondents
Presenter Better
N=30
Copyright 2007, Ida Sim
Outline
• Background
• RCT Schema
»
»
»
modeling approach
the class structure
evaluation
• Relationship to CTO
• Summary
Copyright 2007, Ida Sim
Alignment to CTO
• Have participated in CTO working group from
inception (Simona Carini)
• Contributed RCT Schema classes and definitions to
list of terms for consideration
• Contributed to draft of high-level concept hierarchy
– http://www.bioontology.org/wiki/index.php/Highlevel_Concepts_v0.2
• Contributed to creation of the draft CTO presented
this morning
Copyright 2007, Ida Sim
How Would Trial Bank Use CTO?
• Use CTO as common index into RCT Bank and into
other clinical trial data and information systems
• Map RCT Schema class and slot names to terms in
CTO
»
so that RCT Bank instances can be made available to
machines and humans who wish to
– query
– reason, or
– integrate
»
»
clinical trial information using CTO
e.g., trials co-published with PLoS, etc. into RCT
Bank
Copyright 2007, Ida Sim
Example Use (NCBO)
• Map class names from any trial bank to terms in CTO
»
e.g., in RCT Bank or “European Trial Bank”
– PRIMARY-OUTCOME to CTO term for this
– BASELINE-CHARACTERISTIC to CTO term for this
• CTeXplorer an ontology-driven tool for visualizing
complex design differences across trials [MA Storey, et al
UVic, Canada]
»
»
given variable HgbA1C from any trial bank
would know how to handle and display if annotated as
– PRIMARY-OUTCOME [CTO] or
– BASELINE-CHARACTERISTIC [CTO]
Copyright 2007, Ida Sim
CTO for Integration with
Trial Bank Collaborators...
• National Center for Biomedical Ontology
• Electronic Primary Care Research Network
»
Primary Care Research Object Model
• Global Trial Bank, with AMIA
»
»
trial-bank publishing
“dedicated to assuring the implementation and maintenance
of an open global infrastructure for computable clinical trial
results information”
• European Clinical Trial Data Repository
»
submitted to Framework Programme 7
• with BRIDG, clinical trial management systems, etc?
Copyright 2007, Ida Sim
Summary
• Trial banks are computable repositories of trial information
for analysis, interpretation, and application of trial evidence
to research and care
• We specified the entities for RCT Bank using competency
decomposition method
• RCT Schema implemented entities specification as an
“instance-style” frame-based ontology
»
expressive, extensible, useful for reporting and interpretation
• Mapping of RCT Schema terms to CTO terms makes RCT
Bank instances available for CTO-driven integration and
reasoning
Copyright 2007, Ida Sim