MBSW 2012

Assessing the Impact of Unmeasured Confounding in Comparative Observational Research

Doug Faries Sr Research Scientist, Eli Lilly & Company Based on Work Done by: Karen Price, Xiaomei Peng, Manjiri Pawaskar, James Stamey, John Seaman

Outline

Background • Quality Implementation of Observational Research • Motivation of Unmeasured Confounding Issue Unmeasured Confounding Methods • Focus on: Rule Out / Bayesian Modeling / Multiple Imputation Example Cost Analysis Summary

The Observational Research Problem (or Challenge)

Selection Bias • Physicians/patients did not select treatment ‘at random’ but based on a variety of factors – so Groups A and B differ in some aspects other than treatment Confounders • A variable is a Confounder if it is associated with both treatment selection and outcome

The Observational Research Challenge

Selection Bias • Physicians/patients did not select treatment ‘at random’ but based on a variety of factors – so Groups A and B differ in some aspects other than treatment Confounders • A variable is a Confounder if it is associated with both treatment selection and outcome Measured: Information is collected within the study and statistical adjustment is possible Unmeasured: Information on the confounder is not available from the study

Can We Get Causal Inference From Observational (non-randomized) Data?

YES –

IF

3 key assumptions hold …..

1.

“No Unmeasured Confounders” – An ASSUMPTION! Can not be definitively verified.

2.

“No Perfect Confounding” 3. Correct Models are used

3 4 5 6 1 2

Hierarchy of Evidence

Vandenbrouke (2008), Concato (2000), ….

• Systematic Reviews of Randomized Controlled Trials • Randomized Controlled Trials • Prospective Follow-up Studies • Retrospective Follow-up Studies • Case Control Studies • Anecdotal Case Report and Series

Increasing the Quality of Observational Research (Rubin 2007, 2008)

Keep core statistical (RCT) design principles in mind that are sometimes overlooked in observational research …..

Prospective Specification Multiplicity Replication Sensitivity Analyses

Current State of the Union Regarding Unmeasured Confounding

What should I do about unmeasured confounding?

Current State of the Union Regarding Unmeasured Confounding

What should I do about unmeasured confounding?

Just mention it as a limitation in the Discussion Section and move on!

There are new methods in the literature! “Best Practices” include sensitivity analyses EXPERT

Figure 1: Unmeasured Confounding Options

Unmeasured Confounding Information Available None External Internal Method 1) Rule Out 2) IV 1) Bayesian 2) Algebraic 1) Bayesian 2) Multiple Imputation 3) Propensity Calibration

Example for Today

Pawaskar M, Zagar A, Sugihara T, Shi L (2011). Healthcare resource utilization and costs assessment of type 2 diabetes …... J Med Econ. 2011;14(1):16-27. No direct measure of glycemic control was available in the original claims database. However, after linking with a laboratory file, A1C values were obtained in a subset (about 20%) of the sample; – A1C was a significant predictor of treatment selection (p<.001) but only modestly related to outcome (costs)

Our Work: Sensitivity Analysis using this Internal Information

Information Available: None

Rule Out CONCEPT: Quantify how strong and imbalanced a confounder would need to be in order to explain (‘rule out’) the observed treatment difference Instrumental Variables CONCEPT: Use of an Instrument (variable associated with treatment selection but not with outcome) allows one to mimic randomization

Rule-out Method

Concept: Quantify how strong and imbalanced a confounder would need to be in order to explain (“rule out”) the observed result. This approach attempts to find all combinations of 1) the confounder-outcome relationship and 2) the confounder-treatment relationship, necessary to move the observed point estimate to zero.

Rule Out – Simple Model

Basic (additive) Model: AMD = TTD + Bias • AMD is the apparent (observed) mean treatment difference • TTD is the true (fully adjusted) mean treatment difference • Bias is a function of: – The imbalance of the unmeasured confounding factor between treatment groups – The strength of the association between the unmeasured confounder and the outcome

Rule Out – Simple Spreadsheet

Bias

Pc0 – Pc1

.95

.90

…..

.05

AMD (Estimate)

$ 2,597 $ 2,597

CE

$ 2,734 $ 2,885 $ 2,597 $ 51,940

AMD (Lower 95% CL)

$ 690 $ 690 $ 690

CE

$ 726 $ 767 $ 13,800 Fixed Value 1 Fixed Value 2

Rule-out Method – Example

So, a confounder occurring in 20% more patients in Cohort A (compared to Cohort B) which results in $15,000 higher cost per patient would eliminate the observed difference Trt A is Not Less Costly Trt A remains Less Costly File name/location

Confounder - Outcome Association

Company Confidential Copyright © 2000 Eli Lilly and Company

Rule Out Example

100 90 80 70 60 50 40 30 20 10 0 0,00 AMD (best estimate) AMD (Lower 95% CI)

Confounder / Treatment Relationship (P C1 -P C0 )

1000,00 2000,00 3000,00 4000,00 5000,00 6000,00 7000,00 8000,00 9000,00 10000,00

CE (Confounder Effect on Outcome %)

Information Available: External

Concept: Use information external to the study (e.g. data from literature or other databases) to estimate parameters regarding unmeasured confounding (e.g. strength of association with outcome and treatment).

Bayesian Models: I ncorporate the external information through a prior distribution and account for the uncertainty surrounding the external estimates

External Adjustment (ctd)

Some Issues: 1) Transportability 2) Correlation of Unmeasured Confounder with variables already accounted for in the analysis model Algebraic External Adjustment Examples: - Schneeweiss et al JAGS 2005 - Schneeweiss et al CNS Drugs 2009

Information Available: Internal

Concept: Use information from the patients in the study (e.g. subsample of chart review data for a retrospective claims database study) to estimate parameters regarding unmeasured confounding With Internal data can avoid transportability assumption and can account for correlation between unmeasured confounder and measured confounders

Information Available: Internal

Methods 1) Propensity Score Calibration • Sturmer et al, Am J Epi 2005 2) Bayesian Modeling • McCandless Stat Med 2007 3) Multiple Imputation • Faries (submitted)

Propensity Score Calibration

Concept: Utilize additional data - variables not in full sample but available for a subset of patients - to modify the propensity score adjustment Two propensity scores (PS) are calculated for the validation data, “Error Prone” PS: utilizes only covariates available for the full sample of patients “Gold Standard” PS: calculated using both the covariates in the main study along with the additional confounding covariates.

Regression calibration (measurement error modeling) is then applied to adjust the regression coefficients and thus compensate for the unmeasured confounding.

Propensity Score Calibration (ctd)

Error Prone Propensity Score Model (PS EP )

PS EP



Pr(

X



1 |

z

1

,

z

2

,...,

z n

)

Gold Standard Propensity Score Model (PS GS )

PS GS



Pr(

X



1 |

z

1

,

z

2

,...,

z n

,



)

Calibration Model:

E

[

PS GS

]   0     1

X

    2

PS EP

Company Confidential Copyright © 2000 Eli Lilly and Company File name/location

Propensity Score Calibration

Validity relies on surrogacy of the error prone propensity for the gold standard propensity. • “error prone PS” must be independent of the outcome given “gold standard PS” and treatment.

For our example – surrogacy assumption not clearly satisfied – Correlations of A1C & Outcome was negative – Correlations of Other Covariates & Outcome was positive

Bayesian Twin Regression Models

Concept: Bayesian models naturally incorporate additional sources of information – such as internal subset data or external information from other studies - through prior distributions Outcome   0   1 * Treatment   2 * UnmConf   3 * MeasConf Logit

P

( UnmConf  1 )         

T

reatment     MeasConf Internal data serves in essence as informative prior information for parameters relating to unmeasured confounder Implementation: Win BUGS ( SAS 9.3 code upcoming)

Bayesian Twin Regression Models

Outcome   0   1 * Treatment   * UnmConf   * MeasConf    logit

P

( UnmConf  1 )         

TRT

    MeasConf Priors: Uninformative: Informative:    ,     ,     ,  0 ,  1 ,  2

Keys to Bayesian Approach

• Incorporation of best available information through Informative Priors • Best available data – whether internal (via subset data) or External data (e.g. literature) • Informative Priors – not just a way to add uncertainty (McCandless 2007) • Yields a posterior distribution (point and interval estimates) for the treatment effect adjusted for the unmeasured confounder

U

.

• Fixed Modeling (Schneeweiss 2006) failed to incorporate variability • Flexible data driven model • No restrictions on relationships on associations between variables as in measurement error approaches (Sturmer 2007).

Missing Data Multiple Imputation (for internal data)

Concept: This is a missing data problem – use a well accepted method -- Multiple Imputation! Imputation Model: Treatment, Measured Covariates, and Outcome (in order to account for he association between confounder/outcome and confounder / treatment) Used > 5 replications due to amount of missing data Implementation: PROC MI in SAS

Example: Summary of Sensitivity Analyses with Internal A1C Data

Conclusions

Comparative effectiveness research should include some level of ‘Unmeasured Confounding’ assessment to help consumers of the data understand the robustness of the findings. Bayesian and MI methods are promising approaches - naturally incorporate additional information (internal or external) - can use internal data to avoid development of prior. Lots of Remaining Questions • When is one method preferred to another?

• How much ‘internal data’ is needed for each method?

• When is it cost effective to obtain the internal information as opposed to more easily available external data?

Other References

Goodman M, Barraj LM, Mink PJ, Britton NL, Yager JW, Flanders D, Kelsh MA (2007). Estimating uncertainty in observational studies of associations between continuous variables: example of methylmercury and neuropsychological testing in children Epi Perspec Innovat 4:9.

Gustafson P, McCandless LC (2010). Probabilistic Approaches to Better Quantifying the Results of Epidemiologic Studies. Int. J Environ Res Pub Hlth 7:1520-1539. McCandless, L.C. Gustafson, P. Levy, A.R. Bayesian sensitivity analysis for unmeasured confounding in observational studies. Statist. Med. 2007, 26, 2331-2347.

McCandless, L.C. Gustafson, P. Levy, A.R. A sensitivity analysis using information about measured confounders yielded improved assessments of uncertainty from unmeasured confounding. J. Clin. Epidemiol. 2008, 61, 247 –255.

Pawaskar M, Zagar A, Sugihara T, Shi L (2011). Healthcare resource utilization and costs assessment of type 2 diabetes patients initiating exenatide BID or glargine: a retrospective database analysis. J Med Econ. 2011;14(1):16-27. Epub 2010 Dec 15.

Schneeweiss S, Setoguchi S, Brookhard MA, Kaci L, Wang PS (2009). Assessing Residual Confouding of the Association between Antipsychotic Medications and Risk of Death Using Survey Data. CNS Drugs, 23(2):171-180.

Schneeweiss S, Wang PS (2005). Claims Data Studies of Sedative-Hypnotics and Hip Fractures in Older People: Exploring Residual Confounding Using Survey Information. JAGS 53:948-955. Schneeweiss S, Wang PS (2005). Claims Data Studies of Sedative-Hypnotics and Hip Fractures in Older People: Exploring Residual Confounding Using Survey Information. JAGS 53:948-955.

Sturmer T, Schneeweiss S, Rothman KJ, Avorn J, Glynn RJ (2007). Performance of Propensity Score Calibation – A Simulation Study. Am J Epi 165(10):1110-1118.

Sturmer T, Schneeweiss S, Avorn J, Glynn RJ (2005). Adjusting Effect Estimates for Unmeasured Confounding with Validation Data using Propensity Score Calibration. Am J Epi 162(3):279-289.

Weiner MG, Xie D, Tannen RL (2008). Replication of the Scandinavian Simvastatin Survival Study using a primary care medical record database prompted exploration of a new method to address unmeasured confounding. Pharmacoepi Drug Safety DOI: 10.1002/pds Xue F, Strombom I, Turnbull B, Zhu S, Seeger JD (2011). Duloxetine for Depression and the Incidence of Hepatic Events in Adults. J Clin Psych 31:517-522/