Randomized Experimental Design

What is an Experiment?

• Campbell & Stanley stressed random assignment to experimental treatments.

• I stress manipulation of the independent variable.

• Quasi-Experiments: C&S’s term for research where – there is a manipulated IV – but not random assignment to groups

Random Selection

• Refers to the selection of data points from a population into a random sample.

• This selection procedure will be random if each possible sample of size

N

is equally likely to be sampled.

• Random samples should be representative of the population.

• Our inferential statistics assume random sampling.

Sample AB AC AD BC BD CD Sampling Procedure X Y 1/2 0 1/6 1/6 0 0 0 1/2 1/6 1/6 1/6 1/6 Note. The tabled values are probabilities.

• Y is random, X is not.

• In Y, each time a single score is sampled, all scores in the population are equally likely to be sampled.

Random Assignment

• Refers to the assignment of subjects to treatment conditions.

• Allows us to consider the

populations

(subjects who will get special treatment and those who will not) as equivalent prior to treatment.

• The

samples

will likely differ a little.

Two Basic Randomized Designs

• Randomized Pretest-Posttest Control Group Design

R O X O R O O

• Randomized Posttest Only Control Group Design

R X O R O

Noise Reducing Designs

• These designs reduce noise (error variance) and thus increase power.

Test statistic  treatment effect noise

t

 difference between means standard error of difference

F

 among groups variance error variance

Randomized Blocks Designs

• Matched pairs, randomized blocks, split plot.

• Repeated measures or within-subjects.

• Variance due to the blocking variable is removed from error variance.

Blocks Error Treatment Error Treatment Blocks

Analysis of Covariance

• Change a noise-producing extraneous variable into a covariate that is included in the statistical model.

• Must be able to measure the covariate.

• Variance due to covariate is removed from the error variance.

• Can have more than one covariate.

Factorial ANOVA

• Convert a categorical extraneous variable to an ANOVA factor.

• Variance due to that factor will be removed from the error term.

2 x 2 Factorial Design R R R R X 11 X 12 X 21 X 22 O O O O

Other Randomized Designs Solomon Four Group Design

• Controls threats to internal and external validity as well as the posttest only control group design.

• But has greater power.

• And greater cost – Need more data – More complex analysis

R R R R O O X X O O O O

Solomon Four Group Design

ANOVA

• Arrange all four groups’ posttest scores into a 2 x 2 ANOVA.

• Pretested or Not x Experimental Treatment or Not.

• Significant Interaction – Testing x Treatment threat to External Validity • Main effect of pretesting.

• Main effect of treatment

Treatment effect but no testing or Testing x Treatment interaction Pre-tested no yes marginal Treatment Control X 10 30 10 (10) 30 (10) 10 30 marginal 20 20 20 Pretest means in parentheses

Treatment and testing effects but no Testing x Treatment interaction Pre-tested no yes marginal Treatment none X 10 30 20 (10) 40 (10) 15 35 marginal 20 30 25

Treatment and testing effects and a Testing x Treatment interaction Pre-tested no yes marginal Treatment none X 10 30 20 (10) 60 (10) 15 45 marginal 20 40 30

Solomon Four Group Design

Pretest-Posttest Analysis

• To gain power, analyze the pretest posttest portion of the design with – ANCOV, using pretest scores as covariate – Mixed factorial ANOVA – planned comparisons using

t

• control versus treatment change scores (independent

t

) • pre versus post for control group (correlated

t

) • pre versus post for treatment group (correlated

t

)

Randomized Switching Replications Design

R R O O X O O X O O

• Attempt to control social threats to internal validity.

• Both groups get the special effect, one early, one later.

• May still be social effects with respect to who gets it first.

Switching Replications

Temporary Treatment Effect

Group 1 2 Pretest 150 150 Posttest 1 130 150 Posttest 2 150 130 • Group 1 got the treatment first.

• Treatment is anxiety-reducing drug • DV = measure of anxiety reported by patient

Switching Replications

Persistent Treatment Effect

Group 1 2 Pretest 150 150 Posttest 1 130 150 Posttest 2 130 130 • Treatment is psychotherapy • DV = measure of anxiety reported by patient • A third posttest could show the effect does not last indefinitely.

Switching Replications

Continuing Treatment Effect

Group 1 2 Pretest 150 150 Posttest 1 130 150 Posttest 2 120 130 • Treatment = cognitive psychotherapy • Anxiety continues to decline beyond the first post-treatment observation, as patients get better at employing the cognitive technique.