#### Transcript Planning the Data Analysis: Statistical Power, Mixed

Planning the Data Analysis: Statistical Power, Mixed Effects Linear Models, Moderator and Mediator Models and Related Issues Katie Witkiewitz, PhD Department of Psychology University of New Mexico 2012 NIH Summer Institute on Social and Behavioral Intervention Research July 12, 2012 Ten questions… 1. What is the type of study design? - Number of treatment groups/arms - Randomization procedures - Any clustering? 2. 3. 4. 5. 6. 7. How many subjects can be recruited/observed in a study period? Will there be an equal number of participants in each group? What are the research hypotheses? What do you hope to achieve or learn from each aim? What are the primary outcome measures? What types of variables will be included? - Unit of measurement for DVs, IVs, covariates - Issues with measures (e.g., skewness, counts, inflated zeroes) 8. How many total evaluations and measurements? 9. For repeated measurements, what is the measurement interval? 10. What kinds of missing data patterns do you expect and how much? 2 Other questions depend on design • Qualitative research – – – – – If focus groups, how many groups and group composition? How will data be transcribed and coded? Which software, if any, do you plan to use? What are the research questions? Are you comparing? Aggregating? Contrasting? Sorting? • Secondary data analyses – What is the available data? – Return to questions #4-10 3 Components of the data analysis plan • Study design – – – – Brief overview of design Sampling plan Randomization plan, if applicable Precision/power analysis and sample size • Data management • Statistical analyses – – – – Proposed analyses for each primary aim Secondary or exploratory analyses Interim analyses, if applicable Missing data methodology 4 Study design issues for data analysis • Design issues specific to data analyses – Sampling procedures • Stratified sampling • Clustering – Randomization procedures – # of treatment conditions • Realistically achievable n within each condition • Allocation ratio – Measurement • Unit of measurement for outcome(s) and any covariates • # of measures/constructs • Duration and intervals of assessment – Sample size 5 What sample size do I need? • How many subjects can you realistically recruit? • Precision – how precise do you want to be in measuring the effect? • Power – powering your study to detect a significant effect 6 Precision analyses for pilot studies • Sim & Lewis (2012) recommend n>50 • Precision increases with greater n • In order to carry out any precision-based sample size calculation you need: – Width of confidence interval (e.g., 95%) – narrower interval = more precise estimate – Formula for the relevant standard error 7 Effect of Sample Size on Precision • Estimating a percentage of 30% Sample size 95% CI 50 ± 12.7%, (17.3% - 42.7%) 100 ± 9.0% (21% - 39%) 500 ± 4.0% (26% - 34%) 1000 ± 2.8% (27.2% - 32.8%) 8 Precision Analysis • The sample size required to achieve the desired maximum error E can be chosen as z1 / 2 2 n E 2 2 • Example: suppose that we wish to have a 95% assurance that the error in the estimated mean is less than 10% of the standard deviation (i.e., 0.1). The required sample size is z1 / 2 2 n E 2 2 1 . 96 2 ( 0 . 1 ) 2 2 384 . 2 385 Precision Analysis • Suppose that we wish to have a 95% assurance that the error in the estimated mean is less than 25% of the standard deviation (i.e., 0.25). The required sample size is z 1 / 2 2 n E 2 2 1 . 96 2 2 ( 0 . 25 ) 2 61 . 5 62 Insert Successful, Carefully Controlled Pilot Study Here I have estimates of the effect from my pilot study, now what do I do? 11 Effect size has been defined in various ways • Cohen (1988): “degree to which the phenomenon is present in the population” (p. 9). • Kelley & Preacher (2012): “a quantitative reflection of the magnitude of some phenomenon that is used for the purpose of addressing a question of interest.” (p. 140). • Context of social and behavioral interventions: magnitude of the detectable, minimally expected difference between intervention conditions on outcome of interest. – Standardized difference between two means Cohen’s d x treatment x control S pooled 12 Importance of variability in the achieved power of an intervention 13 Do achieved effect sizes from a prior study = the probable effect size in a subsequent study? Study 1 – Pilot trial d x treatment x control S pooled Study 2 – Main RCT d 10 . 5 9 . 5 0 .5 2 .0 d 10 . 5 9 . 5 0 . 67 1 .5 14 Can I adjust my SD from my pilot to estimate SD for RCT? • If you have pilot data, then you can use the precision analysis logic to “inflate” your observed pilot SD. • Sim & Lewis (2012) provide formulas and calculations 15 Statistical power and sample size • How many subjects do you need to have sufficient statistical power to detect the hypothesized effect? – – – – – What level of power is desirable for your design? What level for statistical significance? What are your statistical tests? What effect size do you suspect to detect? What is the probable variation in the sample? 16 Step-by-step approach to calculating effect sizes. • Decide on the question of interest. – Group differences (Cohen’s d/f2; Hedges g) – Strength of association (r, partial r, β, η2) – Difference in proportion (risk ratio (RR), odds ratio (OR), number needed to treat (NNT)) • Examine prior studies in existing literature to obtain estimates of parameters in the effect size equation. • Find the appropriate formula (or an online calculator) and input the parameters. 17 Online effect size calculators • • • • • www.campbellcollaboration.org/resources/ www.danielsoper.com/statcalc3/ www.divms.uiowa.edu/~rlenth/Power/ http://statpages.org/#Power + many more… 18 Step-by-step approach to power analysis. • Decide on α and β • Using effect size (or range of effect sizes) estimate how many subjects needed to obtain β, given α, for a particular test. • Use existing software or simulation study. 19 Software for estimating power • Within statistics programs: – SAS, R, Stata, SPSS – SamplePower • Stand-alone and online programs: – http://statpages.org/#Power – www.divms.uiowa.edu/~rlenth/Power/ – Optimal Design http://sitemaker.umich.edu/group-based/home – G*power – available free at http://www.psycho.uniduesseldorf.de/aap/projects/gpower/ – PASS –http://www.ncss.com/pass.html – Power & Precision –http://www.power-analysis.com/ – nQuery – www.statistical-solutions-software.com 20 21 22 23 Simulation studies for estimating power • Generate multiple datasets that mimic the design with sample size n and hypothesized parameter estimates as starting values with set effect size – SAS, Mplus, R, Stata, Matlab • More complicated model? Difficulty in determining parameter estimates 24 Data management • Briefly describe your plan for data entry and management. • Overview of preliminary data checking. – Check distributions of primary measures – Examine randomization failures – Test for systematic attrition biases • Clarify how you will handle data issues that may arise. 25 26 Statistical analyses – Types of Analyses – Proposed analyses for each primary aim – Secondary or exploratory analyses • • • • Mediation models Moderation models Mediated-moderation and moderated-mediation Multiple mediator models – Missing data methodology 27 Types of Analyses • Intent-to-treat – includes all randomized subjects, whether or not the subjects were compliant or completed the study • Full analysis set – excludes those who are missing all data by completing no assessments after randomization. May also exclude those who never took a single treatment dose • Per protocol –subset of subjects who complied with the protocol to ensure that the data would be likely to exhibit the effects of the treatment, according to the underlying scientific model 28 Which statistical test? • What are your research hypotheses? • What is the level of measurement for outcome measure(s) and IVs? Outcomes IVs Statistical test Interval/scale 1 DV at 1 time 1 with 2 levels t-test or regression (interval/scale), Mann Whitney (ordinal), χ2 (categorical) 1 with 2+ levels ANOVA or regression (interval/scale), Kruskal-Wallis (ordinal), χ2 (categorical) Interval/scale 2+ DVs at 1 time 1+ with 2+ levels MANOVA, multivariate linear regression, latent variable model Interval/scale repeated measures 1+ with 2+ levels Mixed-effects model (aka random effects regression; multilevel model; latent growth model), repeated measures ANOVA, survival model Categorical DV 1+ with 2+ levels Logistic regression (multinomial if 2+ categories of DV), binary classification test Count DV 1+ with 2+ levels Poisson or negative binomial regression models, generalized linear models 29 30 Latent Variable Models Latent Variable unobserved; unmeasured; based on relationships between observed variables observed variable (x1) observed variable (x2) observed variable (x3) E1 E2 E3 observed; measured variables error or “residual,” not explained by shared variance 31 Latent variables can be continuous or categorical; two representations of the same reality Continuous latent variable – correlation explained by underlying factor Y Ex. structural equation models, factor models, growth curve models, multilevel models X Categorical latent variable – correlation reflects difference between discrete groups on mean levels of observed variables Y X Ex. latent class analysis, mixture analyses, latent transition analysis, latent profile analysis 32 Types of latent variable models Continuous latent variable Categorical latent variable Categorical & continuous latent variable Crosssectional Factor analysis* Latent class/profile analysis Factor mixture model Longitudinal Latent growth curve (i.e., mixed effects*, multilevel, HLM) Latent Markov model (i.e., latent transition analysis) Growth mixture model (i.e., latent class growth, semiparametric group) 33 Factor Analysis • Common tool for examining constructs and creating a measure of related constructs. • Models should be driven by theory, guided by best-practices for model selection, evaluation. 34 Aim #1: Develop a multidimensional measure of alcohol treatment outcome. Analysis plan text for factor analysis model: “Measurement models will be estimated for each construct across studies using a moderated nonlinear factor analysis (MNLFA) approach (Bauer & Hussong, 2009). MLNFA is a novel approach for examining measure structure that allows for items of mixed scale types (i.e., binary, count, continuous) and allows parameters of the factor model to vary as a function of moderator variables (e.g., source of the data, gender).” 35 Mixed Effects Models (aka linear mixed models, random effects regression models, multilevel models, hierarchical linear models, latent growth curve models) • Mixed = “fixed” and “random” effects. • Fixed effects – group level, no variability within individuals (or groups) • Random effects – individual level, variability within individuals (or groups) 36 Mixed effects models in pictures: Fixed effects model Bill Jane Joe y Gordan Sue 1 2 3 time 37 Mixed effects models in pictures: Random-intercept model y Intercept 1 2 3 time Slope Time 1 Time 2 Time 3 X1 X2 X3 ε1 ε2 ε3 38 Mixed effects models in pictures: Random intercept and random slope model y Intercept 1 2 3 time Slope Time 1 Time 2 Time 3 X1 X2 X3 ε1 ε2 ε3 39 Aim #2: To examine the effect of treatment in reducing heavy drinking days among help-seeking alcohol dependent patients. Analysis plan text for mixed effects model: “The primary outcome measure of percent heavy drinking days assessed weekly across 14 weeks of treatment will be examined using a mixed effects model with fixed effects of treatment and random effects of time (week since randomization).” 40 Survival Models • Modeling the amount time (t) to an event (T), where Survival (t) = Pr (T > t) – Comparison in survival rate across groups – Incorporate time-invariant or time-varying covariates • Cox Proportional Hazards Model – Underlying hazard function, h(t)=dS(t)/dt describes how the hazard (risk) changes over time in response to explanatory covariates. 41 Aim #3: To examine the effect of treatment on the amount of time to first drinking or drug use lapse following release from jail. Analysis plan text for survival model: “We will estimate the time to first lapse using a Cox proportional hazards model, where time (t) will be evaluated weekly over the 6-month time interval. The hazard probability for a given week (t) is estimated by the proportion of individuals under observation who are known to have not experienced any drinking or substance use lapses prior to week t that then experienced their first drinking or substance use lapse during week t, conditional on treatment group, gender, and dependence severity.” 42 Testing Mediation • Purpose: To statistically test whether the association between the IV and DV is explained by some other variable(s). • An illustration of mediation a, b, and c are path coefficients. Variables in parentheses are standard errors of those path coefficients. Comparison of Mediation Tests MacKinnon et al 2002 • Baron and Kenny’s approach criticized for having low power - the product of the regression coefficients α and β is often skewed and highly kurtotic • Product of coefficients approach more powerful z’ = α*β/SQRT(β2*sα2 + α2*sβ2) • Bootstrapping to get range of CI for indirect effect: – ProdClin: http://www.public.asu.edu/~davidpm/ripl/Prodclin/ – Macros created for SPSS and SAS: http://www.afhayes.com/spss-sas-and-mplus-macros-andcode.html#sobel – Mplus code: www.statmodel.com – R code generator: http://www.quantpsy.org/medmc/medmc.htm Designing your study to test mediation hypotheses • Key to testing mediation is measurement – Temporal precedence of hypothesized variables in the causal chain • Experimental manipulation, if possible • Mechanism of change? – Mediator ≠ Mechanism – Other steps need to establish mechanism, see Kazdin (2007) 45 What is Moderation? • Variable that affects the direction and/or strength of the relationship between a predictor and a criterion variable – Categorical (e.g., males vs. females) – Continuous (e.g., level of moderator) • Designing your study to test moderation – need a larger sample size More Complex Examples • “Conditional indirect effects” see Preacher, Rucker, & Hayes (2007) – Mediated moderation – Moderated mediation • Latent mediators/moderators • Multiple mediators and/or moderators • Mediation/moderation of growth process Missing Data Issues and Methodology • Design the study to minimize missing data • Acknowledge how you will you examine missing data and test missing data assumptions • Missing data methodologies 48 Examining Missing Data and Missing Data Assumptions • Missing data pattern – which values are missing and which are observed – Univariate missing – confined to a single variable – Monotone missing – missing for all cases, e.g., attrition • Missing data mechanism – why values are missing and the association between missing values and treatment outcomes – Missing completely at random (MCAR) – Missing at random (MAR) – Missing not at random (MNAR) 49 Missing Data Methodologies • Commonly used methods under MAR (or MCAR) 1. 2. Complete case - discard incomplete cases Imputation – fill-in missing values - Single imputation (e.g., mean, baseline, LOCF, BOCF) - Underestimate standard errors and yield biased estimates - Multiple imputation – takes into account uncertainty in imputations 3. Analyze incomplete data using method that does not require complete set - Maximum likelihood - Bayesian methods 50 Problem with single imputation methods 51 Analytic Methods under MNAR • Sensitivity analyses to estimate degree of bias (see Enders, 2010) • Pattern Mixture Models – assume that the substantive data are conditional on the missing data mechanism. The conditional model of the observed data is estimated separately for each missing data pattern. • Selection models – assume that the missing data mechanism is conditional on the substantive data. Observed data are used to predict the probability of missing data. 52 Describing missing data procedures: “For the proposed study we will use maximum likelihood estimation for all analyses, which provides the variance-covariance matrix for all available data and is a preferred method for estimation when some data are missing (Schafer & Graham, 2002). Sensitivity analyses will be used to test the influence of missing data and missing data assumptions (see Witkiewitz et al., 2012).” 53 Common pitfalls 1. 2. 3. 4. 5. Data analysis plan does not match rest of proposal. Cannot address aims or answer hypotheses Not consistent with measures/research design Ignore critical issues or makes unrealistic assumptions. Effect size over-estimated or not included Assumes data will be normally distributed, when unlikely Does not address missing data and attrition biases Measurement invariance when comparing across groups Propose complicated models without statistical expertise in the research team. Propose to test models without clear hypothesis or rationale. Include complex measures (e.g., time varying covariates, genetic information, imaging data), without clear description of how the measures will be analyzed or included in the analyses. 54 References • • • • • • • • • • • Sim & Lewis (2012). The size of a pilot study for a clinical trial should be calculated in relation to considerations of precision and efficiency. J of Clinical Epidemiology, 65, 301-308. Vickers, A. J. (2005). Parametric versus non-parametric statistics in the analysis of randomized trials with non-normally distributed data. BMC Medical Research Methodology, 5, 35. Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Erlbaum. Kelley, K., & Preacher, K. J. (2012). On effect size. Psychological Methods, 17, 137-152. MacKinnon, D. P., Lockwood, C. M., Hoffman, J. M., West, S. G., & Sheets, V. (2002). A comparison of methods to test the significance of the mediated effect. Psychological Methods, 7(1), 83-104. Preacher, K.J., Rucker, D.D., & Hayes, A.F. (2007). Addressing moderated mediation hypotheses: Theory, methods, and prescriptions. Multivariate Behavioral Research, 42, 185-227. Kazdin, A.E. (2007). Mediators and mechanisms of change in psychotherapy research. Annual Review of Clinical Psychology, 3, 1-27. National Research Council. (2010). The Prevention and Treatment of Missing Data in Clinical Trials. Panel on Handling Missing Data in Clinical Trials. Washington, DC: The National Academies Press. Enders, C.K. (2010). Applied missing data analysis. New York: Guilford Press. Schafer, J.L. & J.W. Graham, J. W. (2002). Missing Data: Our View of the State of the Art. Psychological Methods, 7, 147-177. Witkiewitz, K., Bush, T., Magnusson, L. B., Carlini, B. H., & Zbikowski, S. M. (2012). Trajectories of cigarettes per day during the course of telephone tobacco cessation counseling services: A comparison of missing data models. Nicotine and Tobacco Research. doi: 10.1093/ntr/ntr291 55