March 13—Evaluation research and meta
Download
Report
Transcript March 13—Evaluation research and meta
Significance and effect sizes
What is the problem with just using p-levels to determine
whether one variable has an effect on another?
Don’t EVER just give p-range!
Sample results:
For boys, r (87) = .31, p = .03
For girls, r (98) = .24, p = .14
Significance test = effect size x study size
Why are effect sizes important?
What is the difference between statistical, practical, and
clinical significance?
What should you report?
2 group comparison—treatment vs. control on anxiety
symptoms
3 group comparison—positive prime vs. negative
prime vs. no prime on number of problems solved
2 continuous variables—relationship between
neuroticism and goal directedness
3 continuous variables—anxiety as a function of selfesteem and authoritarian parenting
2 categorical variables—relationship between answers
to 2 multiple choice questions
Narrative vs. quantitative reviews
When was the first meta-analysis?
When was the term first used?
What are the advantages of quant reviews?
What are particular critiques of them?
What are the three basic principles to guide meta-
analysis?
Steps to meta-analysis
1. define your variables/question
1 df contrasts
What is a contrast?
2. Decide on inclusion criteria
What factors do you want to consider here?
3. Collect studies systematically
Where do you find studies?
File drawer problem
Rosenthal’s fail-safe N
# studies needed at p < .05= (K/2.706) (K(mean Z squared) = 2.706)
Z = Z for that level of p
K = number of studies in meta-analysis
Funnel plot
Rank correlation test for pub bias
What can you do if publication bias is a problem?
Trim and fill
Sensitivity analysis
Weight studies
Fig. 3. Funnel plots of 11 (subsets of) meta-analyses from 2011 and Greenwald, Poehlman,
Uhlman, and Banaij (2009).
Marjan Bakker et al. Perspectives on Psychological Science
2012;7:543-554
Copyright © by Association for Psychological Science
3. Calculate effect sizes
If there is more than 1 effect per study, what do you do?
What does the sign mean on an effect size?
What are small, medium, and large effects?
How can you convert from one to another?
r or d?
http://www.soph.uab.edu/Statgenetics/People/MBeasl
ey/Courses/EffectSizeConversion.pdf
Families of effect sizes
2 group comparisons (difference between the means)
Cohen’s d
Hedge’s g
Glass’s d or delta
Continuous or multi-group (proportion of variability)
Eta squared η2
Partial eta-squared ηp2
Generalized eta-squared η G2
r, fisher’s z, R2, adjusted R2
ω2 and its parts
difference between η2 and R2 family
Nonparametric effect sizes
Nonnormal data: convert z to r or d
Categorical data:
Rho
Cramer’s V
Goodman-Kruskal’s Lambda
How can you increase your effect sizes?
How can you calculate confidence intervals around your
effect sizes?
http://www.latrobe.edu.au/psy/research/cognitive-and-
developmental-psychology/esci
http://www.cem.org/effect-size-calculator
Interpretation of effect sizes
Recommended for at least most important findings
PS
U
Binomial effect size display (p. 76)
Relative risk
Odds ratio
Risk difference
4. Look at heterogeneity of effect
sizes
Chi-square test
I2 (measure based on Chi-square)
Cochran’s Q
Standard deviations of effect sizes
Stem and leaf plot (p. 671)
Box plot
Forest plot
What are common moderators you might test? How
would you do that?
Forest plot
5. Combine effect sizes
When should you do fixed vs. random effects?
Should you weight effect sizes, and if so, on what?
How can you deal with dependent effect sizes?
Hunter and Schmidt method vs. Hedges et al. method
Credibility intervals vs. confidence intervals
6. Calculate confidence intervals/
7. Look for moderators
What are common moderators you might test?
How do you compare moderators?
“Meta-analysis”
Comparing and combining effect sizes on a smaller
level—when might you want to do this?
How would you do it?
Average within-cell r’s with fisher z transforms
To compare independent r’s: Z = z1-z2/sqrt ((1/n-3) +
(1/n-3))
To combine independent r’s: z = z1+z2/2
Write-up
Inclusion criteria, search, what effect size
Which m-a tech and why
Stem and leaf plots of effect sizes (and maybe mods)
Forest plots
Stats on variability of effect sizes, estimate of pop
effect size and confidence intervals
Publication bias analyses
Side note
Analysis of power (Appendix)
Terms
Evolutionary epistemology
Evidence-based practice
Systems thinking
Dynamical systems approaches
Evaluation research
Issues with evaluation research
What questions are asked?
What methods are used?
What unique issues emerge?
Types of evaluation
Formative
Needs assessment
Evaluability assessment
Structured conceptualization
Implementation evaluation
Process evaluation
Summative
Outcome evaluation
Impact evaluation
Cost-benefit analysis
Secondary analysis
Meta-analysis
Methods used for different ?s
What is the scope of the problem?
How big is the problem?
How should we deliver the program?
How well did we deliver it?
What type of evaluation can we do?
Was the program effective?
What parts of the program work?
Should we continue the program?
Evidence based medicine (Sackett
et al.)
Convert problem into question
Find evidence
Evaluate validity, impact, applicability
Integrate patient experience and clinical judgment
Review evaluation
What does the book author
Mean by an “evaluation culture”?
Is it a good thing?
Post spring break
Readings on analyses (some to be emailed out)
Quant article critique is separate from thought paper
(look for questions at end of syllabus)
One more week then rough drafts due