QUANTITATIVE ANALYSIS

Reporting Standards in Psych

 JARS and MARS  Impetus for them  Covers more than statistics  Use it for your papers in here

Next week

 On Wednesday, give your not-so-rough draft to your reviewer.  Follow the instructions on the peer review sheet .  Use this sheet, the rubric online , and the JARS guidelines to evaluate the paper.

 Meet with your person to go over the feedback.

 Email me your commented-on paper by Friday at 5.

Assignments

 Manny-Stephanie  Melanie-Salomi  Olivia-Lacey  Lee-Tory

Cummings’s suggestions

 Table 1  Provide full report of findings  Preregister study ( Open Science Framework )  Any nonplanned results should be clearly labeled exploratory

How to change to the “new stats”

       Rethink hypotheses—avoid dichotomies Identify what effect you want to measure Prespecify your study Calculate estimates and Cis Make figures using CIs as error bars Interpret ES and CI in writeup Make report and raw data available    http://psychfiledrawer.org/ http://figshare.com/ https://osf.io/

What are threats to conclusion validity? How can you deal with them?

 Low reliability of measures  Problems with your manipulation  “Noise” in the setting  Heterogeneity of participants  Low power (includes those listed above)  Multiple comparisons (fishing and error rate)  Violations of assumptions of your tests  Type I vs. Type II error

Power

 How do you determine power?

 Programs for determining power  http://spark.rstudio.com/jakewestfall/pangea/  G*power  ESCI  Power for NHST vs. accuracy (AIPE)  20 or 50 per group?

 Post hoc power—don’t do it.

Review

 How do these relate to each other?  n  Effect size  ds vs. etas  Alpha  Power  df  One- vs. two-tailed tests

When to use which test

  How do you test relationships vs. levels? GLM (What is it?):         t-test ANOVA ANCOVA MANOVA Regression (what does it mean that it’s least sqs?) Factor analysis (exploratory vs. confirmatory) Multidimensional scaling Discriminant function analysis

What things bias correlations?

 Sample size  Outliers  Truncated range  See page 280 in book—Anscombe’s Quartet

Analysis terms

 Anova factors: random vs. fixed, nested  Methods of regression: stepwise, hierarchical, enter  Types of regression: linear, logistic  Building interaction terms in regression  Multicollinearity  Factor analysis (eigenvalues, exploratory vs. confirmatory, scree plot)

More terms

 Dummy variables (ethnicity in regression)  Regression line  Residuals  Standard errors  Model specification  Least squares

ANCOVAS

 Why can’t you use ANCOVA with nonmanipulated IVs and unadjusted DVs?  Adjusted pretest= mean + reliability(value – mean)  Which reliability estimate should you use?

 Other options:  Matching  Propensity score analysis  Homogeneity of regression assumption

Interpreting confidence intervals

 What is the advantage of using CIs?

 Ways to interpret  Think of what interval means/includes  83% CI for replication ES  Margin of error tells about precision  Cat’s eye graphs  Relate to NHST  Figure 4 (p. 19)  CIs for means vs. differences between them

Steps to analyzing data

    What should be the first thing you do when you begin collecting data?   Even before that:   Design it well to start with Pilot test Check it as you go along What should you do next?  Clean the data Then what?  Examine descriptives And only then  Do inferential stats

Initial checks

  What do you want to look for as you are collecting data?  Make sure people are answering questions correctly  Check for ceiling or floor effects  Check for any other problems or misunderstandings How can these issues be addressed before/during collection?

 Pilot tests  Talk with participants after the study  Have a comment box

What checks should occur during data entry/cleaning?

         Do double entry or random checks Do frequencies to look for duplicates or impossible scores Check IP addresses and time stamps Declare missing values (and make sure it can’t be a real value) Add value labels and label all variables Do recodes in syntax or into new variables Do computes in syntax and describe them with value labels Make a participant variable and variables for when/where collected, by whom, time started and ended, etc.

Clearly label your datafiles so you know which is which (make a codefile with info on the study and data)

Other common data management problems

   When you recode categories, make sure your recodes make sense (e.g., political orientation) Do reliability item analyses to check for poor items and to see if there are ones that should have been recoded and weren’t Figure out how to appropriately deal with missing values     Little’s MCAR test (not available in our version) Imputation Re-read our earlier article on missing values!

What is the default in SPSS and why is that bad?

What else should you do before you begin analyzing your data?

 Get to know your data!!!

     Do frequencies and look at the distributions and ranges Look at the means and SDs and think about what those really mean (measures of central tendency and variability)  Std deviation vs. variance vs. std error Do crosstabs (what test would go with these?) Do scatterplots and other graphs  Histogram    Stem and leaf Boxplot Check out Edward Tufte’s books Look for outliers—why do you get them and how can you find them?

Check for violations of assumptions

      Normality   Univariate and multivariate Skew and kurtosis Linearity    Multivariate Scatterplots Graphing residuals Homoscedasticity    Homogeneity of variance at univariate level (Levene’s test) Scatterplots Box’s M at multivariate level Use analyze/descriptives/explore or within GLM options Independence (more on that later) May need to transform variables

Review

 Mediator  Moderator  Covariate  Control variable  How many effects are there in a 2 x 2 x 3 design? How many people for n=10 with a between, between, within design?

 What are df and how/when do you report them?

Other types of analyses

 SEM, HLM

Issues of nonindependence

 Why might data not be independent? Examples of dyad and group analyses in different areas of psych?

 Why should we care? When is it a nuisance vs. important for its own sake?

 How can you determine the degree of nonindep?

 Is there a natural distinction in dyads? Then r  If not, or if groups, intraclass correlation  What is an intraclass correlation? What does it tell you?  How can you test its significance?

ICCs

   Calculation: www.uni.edu/harton/ICC.xlsx

ICC sig. test has low power, so use liberal test ( a = .20) Effects of nonindependence depends on direction and type of IV (table 17.5)  Examples of  Between IV  Within IV  Mixed IV (not design!)

Biases in ICC

 When between variable and negative ICC, too conservative  When between and positive ICC, too liberal  (opposite with within, differs with mixed)

How to deal with nonindep

   Between variables: 3 levels of variation: A, G/A, and S/G/A   Can use group or individual level depending, using different F denom Regression for continuous data Within variables: 4 levels of variation: A, G, G x A, S/G x A Mixed variables: Actor-partner interdependence model   Within dyad regression (differences) then between dyad regression (averages) Regression coefficients used to estimate actor and partner effects

When multi-level models are needed

    The lower level n is not the same for every upper level You’re interested in interactions between lower and upper level variables The lower levels within upper levels are potentially nonindependent Analysis:   Regression for each upper-level group, then use upper-level predictors to predict intercepts and slopes of groups (wted) Doesn’t work if negative ICC and problematic for dyads/small groups (need group size > (n lower level predictors + 2)

When there are multiple dyads in a group

 Round robin designs  Block designs  Kenny’s social relations model (actor-ptr indep. Model)  Components: John’s rating of Ashley = average for the class, John’s tendency to see people as productive, Ashley’s tendency to be seen as productive, and John’s unique view of Ashley (group mean, actor, partner, and relationship effects)  Can also test correlations between actor and partner or with self-reports

Lab assignment

 Check your email