Transcript Experimental Design, Statistical Analysis

Experimental Design,
Statistical Analysis
CSCI 4800/6800
University of Georgia
March 7, 2002
Eileen Kraemer
Research Design
Elements:





Observations/Measures
Treatments/Programs
Groups
Assignment to Group
Time
Observations/Measure
Notation: ‘O’

Examples:
 Body weight
 Time to complete
 Number of correct response
Multiple measures: O1, O2, …
Treatments or Programs
Notation: ‘X’




Use of medication
Use of visualization
Use of audio feedback
Etc.
Sometimes see X+, X-
Groups
Each group is assigned a line in the
design notation
Assignment to Group
R = random
N = non-equivalent groups
C = assignment by cutoffs
Time
Moves from left to right in diagram
Types of experiments
True experiment – random assignment
to groups
Quasi experiment – no random
assignment, but has a control group or
multiple measures
Non-experiment – no random
assignment, no control, no multiple
measures
Design Notation Example
R
O1
R
O1
X
Pretest-posttest treatment
comparison group
randomized experiment
O1,2
O1,2
Design Notation Example
N
O
N
O
X
Pretest-posttest
Non-Equivalent Groups
Quasi-experiment
O
O
Design Notation Example
X
Posttest Only
Non-experiment
O
Goals of design ..
Goal:to be able to show causality
First step: internal validity:
If x, then y
AND
 If not X, then not Y

Two-group Designs
Two-group, posttest only, randomized
experiment
R
R
X
O
O
Compare by testing for differences between means
of groups, using t-test or one-way Analysis of
Variance(ANOVA)
Note: 2 groups, post-only measure, two distributions each
with mean and variance, statistical (non-chance) difference
between groups
To analyze …
What do we mean
by a difference?
Possible Outcomes:
Measuring Differences …
Three ways to estimate effect
Independent t-test
One-way Analysis of Variance (ANOVA)
Regression Analysis (most general)
equivalent
Computing the t-value
Computing the variance
Regression Analysis
Solve overdetermined system of equations for β0 and
β1, while minimizing sum of e-terms
Regression Analysis
ANOVA
Compares differences within group to
differences between groups
For 2 populations, 1 treatment, same as
t-test
Statistic used is F value, same as square
of t-value from t-test
Other Experimental Designs
Signal enhancers

Factorial designs
Noise reducers


Covariance designs
Blocking designs
Factorial Designs
Factorial Design
Factor – major independent variable

Setting, time_on_task
Level – subdivision of a factor


Setting= in_class, pull-out
Time_on_task = 1 hour, 4 hours
Factorial Design
Design notation as
shown
2x2 factorial design
(2 levels of one
factor X 2 levels of
second factor)
Outcomes of Factorial Design
Experiments
Null case
Main effect
Interaction Effect
The Null Case
The Null Case
Main Effect - Time
Main Effect - Setting
Main Effect - Both
Interaction effects
Interaction Effects
Statistical Methods for
Factorial Design
Regression Analysis
ANOVA
ANOVA
Analysis of variance – tests hypotheses
about differences between two or more
means
Could do pairwise comparison using ttests, but can lead to true hypothesis
being rejected (Type I error) (higher
probability than with ANOVA)
Between-subjects design
Example:




Effect of intensity of background noise on
reading comprehension
Group 1: 30 minutes reading, no
background noise
Group 2: 30 minutes reading, moderate
level of noise
Group 3: 30 minutes reading, loud
background noise
Experimental Design
One factor (noise), three levels(a=3)
Null hypothesis: 1 = 2 = 3
Noise
None
Moderate
High
R
O
O
O
Notation
If all sample sizes same, use n, and
total N = a * n
Else N = n1 + n2 + n3
Assumptions
Normal distributions
Homogeneity of variance

Variance is equal in each of the populations
Random, independent sampling
Still works well when assumptions not
quite true(“robust” to violations)
ANOVA
Compares two estimates of variance


MSE – Mean Square Error, variances within
samples
MSB – Mean Square Between, variance of
the sample means
If null hypothesis


is true, then MSE approx = MSB, since
both are estimates of same quantity
Is false, the MSB sufficiently > MSE
MSE
MSB
Use sample means to calculate sampling
distribution of the mean,
=1
MSB
Sampling distribution of the mean * n
In example, MSB = (n)(sampling dist) =
(4) (1) = 4
Is it significant?
Depends on ratio of MSB to MSE
F = MSB/MSE
Probability value computed based on F value,
F value has sampling distribution based on
degrees of freedom numerator (a-1) and
degrees of freedom denominator (N-a)
Lookup up F-value in table, find p value
For one degree of freedom, F == t^2
Factorial Between-Subjects
ANOVA, Two factors
Three significance tests



Main factor 1
Main factor 2
interaction
Example Experiment
Two factors (dosage, task)
3 levels of dosage (0, 100, 200 mg)
2 levels of task (simple, complex)
2x3 factorial design, 8 subjects/group
Summary table
SOURCE
Task
Dosage
TD
ERROR
TOTAL
df Sum of Squares
1
47125.3333
2
42.6667
2
1418.6667
42
5152.0000
47
53738.6667
Sources of variation:




Task
Dosage
Interaction
Error
Mean Square
F
p
47125.3333 384.174 0.000
21.3333
0.174 0.841
709.3333
5.783 0.006
122.6667
Results
Sum of squares (as before)
Mean Squares = (sum of squares) /
degrees of freedom
F ratios = mean square effect / mean
square error
P value : Given F value and degrees of
freedom, look up p value
Results - example
Mean time to complete task was higher
for complex task than for simple
Effect of dosage not significant
Interaction exists between dosage and
task: increase in dosage decreases
performance on complex while
increasing performance on simple
Results