Chapter 10 – Lecture 10 Internal Validity – Control through Experimental Design 1) Test the effects of IV on DV 2) Protects against threats.
Download ReportTranscript Chapter 10 – Lecture 10 Internal Validity – Control through Experimental Design 1) Test the effects of IV on DV 2) Protects against threats.
Chapter 10 – Lecture 10
Internal Validity – Control through Experimental Design 1) Test the effects of IV on DV 2) Protects against threats to internal validity Causation
Experimental Design Highest Constraint Comparisons btw grps Random sampling Random assignment Infer Causality
Experimental Design (5 characteristics) 1) One or more hypothesis 2) Includes at least 2 “levels” of IV 3) Random assignment 4) Procedures for testing hypothesis 5) Controls for major threats to internal validity
Experimental Design Develop the problem statement Define IV & DV Develop research hypothesis Identify a population of interest Random sampling & Random assignment Specify procedures (methods) Anticipate threats to validity Create controls Specify Statistical tests • Ethical considerations Clear Experimental Design…
Experimental Design 2 sources of variance 1. between groups variance (systematic) no drug drug 2. Within groups variance (nonsystematic) (error variance) Remember… Sampling error Significant differences…variability btw means is larger than expected on the basis of sampling error alone (due to chance alone)
Variance VARIANCE “Partitioning of the variance” Need it!
Without it… No go Between Group Experimental Variance (Due to your treatment) + Extraneous Variance (confounds etc.) CON TX Within Group Error Variance (not due to treatment – chance) Subs
Variance: Important for the statistical analysis
F =
between groups variance Within groups variance
F =
Systematic effects + error variance error variance
F =
1.00
No differences btw groups
Variance Your experiment should be designed to • Maximize experimental variance •Control extraneous variance •Minimize error variance
Maximize “Experimental” Variance • At least 2 levels of IV (IVs really vary?) •Manipulation check: make sure the levels (exp. conditions) differ each other Ex: anxiety levels (low anxiety/hi anxiety) performance on math task anxiety scale
Control “Extraneous” Variance 1. Ex. & Con grps are similar to begin with 2. Within subjects design (carryover effects??) 3. If need be, limit population of interest (o vs o ) 4. Make the extraneous variable an IV (age, sex, socioeconomic) = factorial design M F Lo Anxiety M-low F-low
Factorial design (2 IV’s)
Hi Anxiety M-hi F-hi
YOUR Proposals
Control through Design – Don’ts 1. Ex Post Facto 2. Single-group, posttest only 3. Single-group pretest-posttest 4. Pretest-Posttest natural control group 1. Ex Post Facto – “after the fact” Group A Naturally Occurring Event Measurement No manipulation
Control through Design – Don’ts Single group posttest only Group A TX Posttest Single group Pretest-posttest Pretest Group A TX Posttest Compare
Control through Design – Don’ts Pretest-Posttest Naturalistic Control Group Group A Pretest TX Posttest Compare Group B Pretest no TX Posttest Natural Occurring
Control through Design – Do’s – Experimental Design • Manipulate IV • Control Group • Randomization 4 Basic Designs Testing One IV 1. Randomized Posttest only, Control Group 2. Randomized Pretest-Posttest, Control Group 3. Multilevel Completely Randomized Between Groups 4. Solomon’s Four- Group
Randomized Posttest Only – Control Group (most basic experimental design) R Group A TX Posttest (Ex) R Group B (Con) no TX Posttest Compare
Randomized, Pretest-Posttest, Control Group Design R Group A Pretest TX Posttest (Ex) R Group B (Con) Pretest no TX Posttest Compare
Multilevel, Completely Randomized Between Subjects Design (more than 2 levels of IV) R Group A Pretest TX1 Posttest R Group B Pretest TX 2 Posttest R Group C Pretest TX3 Posttest R Group D Pretest TX4 Posttest Compare
Solomon’s Four Group Design (extension Multilevel Btw Subs) R Group A Pretest TX Posttest R Group B Pretest --- Posttest R Group C ------- R Group D ------- TX Posttest --- Posttest Powerful Design!
Compare
What stats do you use to analyze experimental designs? Depends the level of measurement Test difference between groups Nominal data chi square (frequency/categorical) Ordered data Mann-Whitney U test Interval or ratio t-test / ANOVA (F test)
t-Test Compare 2 groups Independent Samples (between Subs) Evaluate differences bwt 2 independent groups One sample (Within) Evaluate differences bwt two conditions in a single groups
Assumptions to use t-Test 1. The test variable (DV) is normally distributed in each of the 2 groups 2. The variances of the normally distributed test variable are equal – Homogeniety of Variance 3. Random assignment to groups
t-distribution Represents the distribution of t that would be obtained if a value of t were calculated for each sample mean for all possible random samples of a given size from some population
Degrees of freedom (df) When we use samples we approximate means & SD to represent the true population Sample variability (SS = squared deviations) tends to underestimate population variability Restriction is placed = making up for this mathematically by using n-1 in denominator
S
2
= variance ss (sum of squares) df (degrees of freedom) (x n-1
x
)
2 Degrees of freedom (df): n-1 The number of values (scores) that are free to vary given mathematical restrictions on a sample of observed values used to estimate some unknown population = price we pay for sampling
=x X 8 ?
6 Degrees of freedom (df): n-1 Number of scores free to vary Data Set you know the mean (use mean to compute variance) n=2 with a mean of 6 In order to get a mean of 6 with an n of 2…need a sum of 12…second score must be 4… second score is restricted by sample mean (this score is not free to vary)
ENDURANC DRUG doped no dope
Group Statistics
N 10 10 Mean 7.9000
2.6000
Std. Deviation 1.1972
1.2649
Std. Error Mean .3786
.4000
Independent Samples Test
ENDURANC Equal variances assumed Equal variances not assumed Levene's Test for Equality of Variances F .065
Sig.
.801
t 9.623
9.623
t-test for Equality of Means df 18 Sig. (2-tailed) Mean Difference .000
5.3000
Std. Error Difference .5508
95% Confidence Interval of the Difference Lower Upper 4.1429
6.4571
17.946
.000
5.3000
.5508
4.1427
6.4573
ANOVA
ENDURANC Between Groups Within Groups Total Sum of Squares 140.450
27.300
167.750
df 1 18 19 Mean Square 140.450
1.517
F 92.604
Sig.
.000
Analysis of Variance (ANOVA) Two or more groups ….can use on two groups…
t
2 = F Variance is calculated more than once because of varying levels (combo of differences) Several Sources of Variance SS – between SS – Within SS – Total Partitioning the variance Sum of Squares: sum of squared deviations from the mean
Assumptions to use ANOVA 1. The test variable (DV) is normally distributed 2. The variances of the normally distributed test variable is equal – Homogeniety of Variance 3. Random assignment to groups
F =
between groups variance Within groups variance
F =
Systematic effects + error variance error variance
F =
1.00
No differences btw groups F = 21.50
22 times as much variance between the groups than we would expect by chance
After Omnibus F… Planned comparisons & Post Hoc tests A Priori (spss: contrast) A Posteriori part of your hypothesis…before data are collected…prediction is made Not quite sure where differences will occur
Why not just do t-tests!
2 types of errors that you must consider when doing Post Hoc Analysis Alpha 1. Per-comparison error (PC) 2. Family wise error (FW) Inflate Alpha!!!!
1 vs 2 1 vs 3 1 vs 4 1 vs 5 2 vs 3 2 vs 4 2 vs 5 3 vs 4 3 vs 5 4 vs 5
FW = c(
)
c = # of comparisons made
= your PC Ex: IV ( 5 conditions)
FW = c(
) 10 (0.05) = .50
HSD