Transcript Etched01” Title Slide

© 2003 By Default!
Slide 1
Physician Prognostic
Accuracy for In-Hospital
Mortality in Percutaneous
Coronary Intervention
Michael E. Matheny, MD
Medical Informatics Fellow
Decision Systems Group
Brigham & Women’s Hospital
Boston, MA
A Free sample background from www.powerpointbackgrounds.com
© 2003 By Default!
Slide 2
Specific Aims

Primary Hypothesis
– Accuracy of subjective physician estimations of
in-hospital mortality will be similar or improved
when compared with accepted objective risk
assessment methods for percutaneous coronary
intervention (PCI)
A Free sample background from www.powerpointbackgrounds.com
© 2003 By Default!
Slide 3
Specific Aims

Secondary Hypotheses
– Accuracy of subjective physician estimations of inhospital major adverse cardiac events (MACE) will be
similar or improved when compared with accepted
objective risk assessment methods for PCI
– Qualitative collection of risk factors could identify
additional important risk factors currently not included in
the objective risk models
– Incorporating subjective physician estimates into an
objective risk model will outperform either separately
A Free sample background from www.powerpointbackgrounds.com
© 2003 By Default!
Slide 4
Background

Discrimination
– Ability to predict an outcome on a population level
– Area under the Receiver Operating Characteristic Curve
(AUC)
A Free sample background from www.powerpointbackgrounds.com
© 2003 By Default!
Slide 5
Background

Calibration
– Ability to predict an outcome on a case/small group level
– Hosmer-Lemeshow Goodness-of-Fit Test (HL-GF)
– Brier Score
A Free sample background from www.powerpointbackgrounds.com
© 2003 By Default!
Slide 6
Background

Subjective vs APACHE II Medical ICU
Mortality 1
– Discrimination:
Objective Better
– Calibration:
Subjective Better
– Forecasting Improves with Training

Subjective vs APACHE II Medical ICU
Mortality 2
– Discrimination:
– Calibration:
A Free sample background from www.powerpointbackgrounds.com
Subjective Better
No Difference
© 2003 By Default!
Slide 7
Background

Subjective vs LR Acute CHF 90 day and 1
year Mortality 3 4
– Discrimination:
– Calibration:
– All estimations poor

No Difference
No Difference
Subjective vs SNAP Neonatal ICU Mortality 5
– Discrimination:
– Calibration:
A Free sample background from www.powerpointbackgrounds.com
No Difference
No Difference
© 2003 By Default!
Slide 8
Background

Subjective + PRISM III Pediatric ICU
Mortality 6
– Discrimination:
– Calibration:
– Combined model
• Discrimination:
• Calibration:
A Free sample background from www.powerpointbackgrounds.com
No Difference
No Difference
Improved from either
Improved from either
© 2003 By Default!
Slide 9
Background

Subjective vs LR Model of Post-Op mortality
for Open Heart Surgeries 7
– Discrimination:
– Calibration:
– Combined model
• Discrimination:
• Calibration:
No Difference
No Difference
No Difference
No Difference
– Subjective assessments were more calibrated at
the extremes of probability
A Free sample background from www.powerpointbackgrounds.com
© 2003 By Default!
Slide 10
Background

Subjective Physician Assessments
– Multiple Forms of Bias 8
•
•
•
•
Ego Bias
Regret
Ignoring Negative Evidence
Framing
A Free sample background from www.powerpointbackgrounds.com
© 2003 By Default!
Slide 11
Background

No work has been done evaluating
subjective physician estimates for in-hospital
mortality in percutaneous coronary
interventions.
A Free sample background from www.powerpointbackgrounds.com
© 2003 By Default!
Slide 12
Background
PCI Objective Risk Model Gold Standards
 Logistic Regression
• National
Models
– American College of Cardiology
– 50123 pts 1998 - 2000
• Regional
– Northern New England 10
– 15331 pts 1994 - 1996
• Local
– Brigham & Women’s Hospital 11
– 2804 pts 1997 - 1999
A Free sample background from www.powerpointbackgrounds.com
9
© 2003 By Default!
Slide 13
Background
Pilot Data
 Recent
Evaluation of Models on Local
Institution Data 12
– Discrimination (AUC)
• ACC 0.90
• NNE 0.89
• BWH 0.89
– Calibration (HL-GF)
• ACC <0.001
• NNE <0.001
• BWH <0.001
A Free sample background from www.powerpointbackgrounds.com
© 2003 By Default!
Slide 14
Background

Objective Assessment Models
– Multiple Forms of Bias
• Population/Demographic Bias
– Regional Variances
• Selection Bias
– Population referral bias
• Temporal Bias
– Medical Care Standards
– Data Documentation
• Data Noise
– Heterogeneous Data Standards
– Variation in Data Element Collection
– Data Entry Quality Variations
A Free sample background from www.powerpointbackgrounds.com
© 2003 By Default!
Slide 15
Background

Incomplete model information?

Best Described Risk Factors
Age
Sex
Ejection Fraction
Recent MI
Hemodynamic Stability
Intra-Aortic Balloon Pump
A Free sample background from www.powerpointbackgrounds.com
CHF
Unstable Angina
Renal Failure
Hx (CAD, DM, COPD, HTN)
Prior CABG or PCI
© 2003 By Default!
Slide 16
Study
Design

Prospective Cohort Study
A Free sample background from www.powerpointbackgrounds.com
© 2003 By Default!
Slide 17
Study
Population

Location

Inclusion Criteria

Exclusion Criteria
– Brigham & Women’s Interventional Cardiology
Suites
– All Patients presenting for pre-operative
evaluation for PCI
– Procedural Team declines to participate in survey
A Free sample background from www.powerpointbackgrounds.com
© 2003 By Default!
Slide 18
Study
Data Collection

Paper Survey

Subjective mortality assessment (0-100%)
before and after procedure
– Administration refused to allow survey to be part
of medical record
– Attendings
– Fellows
– Scrub Nurse

Qualitative additional risk factors from
Attendings
A Free sample background from www.powerpointbackgrounds.com
© 2003 By Default!
Slide 19
Exposures

Percutaneous Coronary Transluminal
Angiography with or without Coronary
Stenting
A Free sample background from www.powerpointbackgrounds.com
© 2003 By Default!
Slide 20
Covariates/Confounders
Age
Sex
Ejection Fraction
Recent MI
Hemodynamic Stability
Intra-Aortic Balloon Pump
A Free sample background from www.powerpointbackgrounds.com
CHF
Unstable Angina
Renal Failure
Hx (CAD, DM, COPD, HTN)
Prior CABG or PCI
© 2003 By Default!
Slide 21
Outcomes
In-Hospital Death
 In-Hospital MACE

A Free sample background from www.powerpointbackgrounds.com
© 2003 By Default!
Slide 22
Analysis Plan

Measure Discrimination & Calibration on local data
for:
– Objective MACE & Mortality Models
• National
• Regional
• Local
– Subjective MACE & Mortality “Models”

Pair-wise Comparison of Objective and Subjective
models for statistical differences

Develop LR model with subjective data as a
covariate, and perform pair-wise comparisons with
objective and subjective models to determine if new
model shows improvement
A Free sample background from www.powerpointbackgrounds.com
© 2003 By Default!
Slide 23
Analysis Plan

Local Institution Data
– ~1% Death Rate
– ~5% MACE Rate
– ~200 Cases / month
A Free sample background from www.powerpointbackgrounds.com
© 2003 By Default!
Slide 24
Analysis Plan
Rough Guess

Sample Size Calc
– Binomial Fisher’s Exact
– Α = 0.05
– Power = 0.80
– Effect Size & Estimated Sample
• Mortality
– 1% to 1.5% = 8150
– 1% to 2% = 2514
• MACE
– 5% to 7.5% = 1550
– 5% to 10% = 473
A Free sample background from www.powerpointbackgrounds.com
© 2003 By Default!
Slide 25
Analysis Plan
Recruitment

Multi-Center

No Termination Date
– Exploring recruitment possibilities from BethIsrael and Massachusetts General Cath Labs
– Implemented as Quality Control method in BWH
Cath Lab
A Free sample background from www.powerpointbackgrounds.com
© 2003 By Default!
Slide 26
Limitations

Sample Size

Paper Survey

Study Population Compliance

Selection Bias
A Free sample background from www.powerpointbackgrounds.com
© 2003 By Default!
Slide 27
Time Table

IRB Approval

Physician Survey Template

IC Lab Tech Data Collection Training

Data Collection
– Completed
– August 2005
– September 2005
– September 2005 - Open
A Free sample background from www.powerpointbackgrounds.com
© 2003 By Default!
Slide 28
Acknowledgements

Co-Authors

Funding
– Nipun Arora, MD
– Lucila Ohno-Machado, MD, PhD
– Frederic S. Resnic, MD, MS
– NLM 1-T15-LM-07092
A Free sample background from www.powerpointbackgrounds.com
© 2003 By Default!
Slide 29
The End
[email protected]
Michael Matheny, MD
Brigham & Women’s Hospital
Thorn 309
75 Francis Street
Boston, MA 02115
A Free sample background from www.powerpointbackgrounds.com