Transcript Scheffe Post Hoc

Scheffé Post-Hoc
Comparisons
Advanced Research Methods in
Psychology
- lecture -
Matthew Rockloff
1
When to use a Scheffé Posthoc
comparison … 1



Whenever an ANOVA model is used to
examine the differences among more than 2
groups, a posthoc procedure can be used to
compare differences between all pairs of
means.
Posthoc comparisons are very similar
to t-tests.
However, posthoc comparisons are more
appropriate for multiple tests, because they
help control type-I error.
2
When to use a Scheffé Posthoc
comparison … 2

Type I-error is the chance of wrongly
accepting differences between
means as significant.

In a t-test, this chance is controlled
to be at most 5%.
In other words, we only accept 2
means as significantly different if the
p-value in the t-test is less than “.05.”

3
When to use a Scheffé Posthoc
comparison … 3



In a study that has several groups, we could
do several t-tests to compare all the
differences between means.
However, each t-test has a separate 5%
chance of making a wrong conclusion by
falsely declaring 2 means significantly
different.
When we do several tests, the chance of
making at least one wrong conclusion starts
to expand dramatically.
4
When to use a Scheffé Posthoc
comparison … 4



To understand “why” consider a simple toss
of a fair coin.
Each toss has a 50% chance of yielding
“heads.”
What if we tossed the coin 100 times?
 What is the chance of having
at least one “heads?”
 Pretty darn likely!
5
When to use a Scheffé Posthoc
comparison … 5



In the same way, the more t-tests we
perform, the more likely we’ll get at least one
conclusion wrong.
Fortunately, posthoc comparisons have
been con structured to adjust for this
problem.
They are more conservative than t-tests and
control for Type-I error.
6
Example 8.1



We’ll return once again to the diet example.
However, this time there are 3 different diets;
including pizza, beer and cream.
The outcome is weight gain.
The research question follows:
“Is there any difference in
weight gain between the 3 diets?”
7
Example 8.1 (cont.)



The first step is to create an
ANOVA table similar to our
previous example.
See last week’s example for a
detailed explanation of how to
calculate a Oneway ANOVA.
However, a brief hand-worked
solution is provided on the next
slide 
8
Pizza
1
2
2
2
3
Weight Gains:
Beer
3
4
4
4
5
Cream
5
5
6
7
7
2
0.4
4
0.4
6
0.8
 j 
j 
S2xj
 j 
=
S between

2
S
2
between
2
S within 

S within
2
2 St 
t
2
S
N=
n=

S
2
between

2

S within
St 
2
2.667
0.533
3.2
15
5
Calculations continued next slide 
9
Example 8.1 (cont.)
ANOVA table
Degrees
Source of Sum of of
Mean
Critical Reject
Variance Squares Freedom Square F-ratio
Value
Decision
SV
SS
df
MS
F
CV
Reject?
BTW
40
2
20
30
3.89 Yes
WITH
8
12
0.667
TOT
48
14
10
Example 8.1 (cont.)


Before we can do posthoc comparisons, we
must interpret the ANOVA table.
Since we now have 3 conditions, the
conclusion we make is slightly different:
There was model significance for the ANOVA,
F(2,12) = 30.00, p < .05, indicating at least
one significant difference among the means.
11
Example 8.1 (cont.)





The ANOVA table provides us with what is called the
“omnibus F-test” or likewise “model significance.”
In the Oneway ANOVA, this test shows that there is
at least one significant difference between a pair of
means.
Of course, there are 3 unique pairs of means in our
example.
The omnibus F-test does not indicate which pairs
are significantly different.
Therefore, we need a posthoc comparison to
examine the difference between all pairs.
12
The Scheffé posthoc
calculation by hand




There are several types of posthoc comparisons,
most of which are named after dead statisticians
(e.g., Tukeys HSD, Fishers LSD and Ducans Multiple
Range Test).
Scheffé is the most conservative of the bunch.
It controls experiment wide error rate to 5%, or in
other words, there at most a 5% chance of making
any type-I errors using the procedure.
While this is desirable, it also makes the Scheffé
procedure less sensitive.
13
The Scheffé posthoc
calculation by hand (cont.)



Using Scheffé, we are more exposed to
making the Type-II error of not identifying
significant difference when they do exist.
Other procedures make a different tradeoff
between exposure to type-I and type-II error.
As such, none is “better” than the other,
only different in exposures to this natural
tradeoff.
14
The Scheffé posthoc
calculation by hand (cont.)

The first step to using the
Scheffé procedure involves
calculating the differences
in all pairs of means:

In our example:
L1 = 1   2
L1 = -2
L2 =   
1
3
L2 = -4
L3 =
L3 = -2
2  3
15
The Scheffé posthoc
calculation by hand (cont.)

Next, we need to compare these differences
to Scheffé’s critical “S” value:
S
( J  1)  F  2  (
MS within
n
)
,
where Fα is the critical
value of the ANOVA table
(see previous calculation).
16
The Scheffé posthoc
calculation by hand (cont.)

In our example, the critical “S” is:
S
(3  1)  3.89  2  (
0.667
5


) = 1.44
Since the absolute value of each difference (L1, L2
and L3) exceeds the critical value, we can declare
significant difference between all pairs of means.
Summarizing our findings from the ANOVA and the
posthoc comparisons, we conclude: (see next slide)
17
We conclude …
There was model significance for the ANOVA,
F(2,12) = 30.00, p < .05, indicating at least one
significant difference among the means. Scheffé
posthoc comparisons showed that weight gain
was higher in the Cream condition (M = 6.00) than
the Beer condition (M = 4.00), p < .05 (twotailed). In turn, weight gain was lower in the Pizza
condition (M = 2.00) than either the Beer
Condition, p < .05 (two-tailed), or the Cream
condition, p < .05 (two-tailed).
18
Alternative conclusion …

The conclusion may still be somewhat hard
to read. Therefore, an alternative conclusion
can refer the reader to a figure that
illustrates the differences:

See next two slides
19
Alternative conclusion: pt 1 …
There was model significance for the ANOVA,
F(2,12) = 30.00, p < .05, indicating at least one
significant difference among the means. In
addition, a Scheffé posthoc comparison showed
that all means were significantly different, p < .05
(two-tailed). Weight gain was highest in the Cream
condition, followed by the Beer condition and the
Pizza condition (see figure 8.1).
(See next slide.)
20
Alternative conclusion: pt 2 …
Weight Gain
Figure 8.1: Weight Gain by Diet
8
7
6
5
4
3
2
1
0
Pizza
Beer
Cream
Pizza
Beer
Cream
Diet
21
Example 8.1 Using SPSS
As before, we need to setup variables in SPSS:
 IndependentVariable = diet (1 = Pizza, 2 = Beer,
3 = Cream)
 DependentVariable = wtgain (weight gain)
22
Example 8.1
Using SPSS (cont.)
The data is
entered into the
SPSS data view
as demonstrated
opposite 
23
Example 8.1 Using SPSS (cont.)

The SPSS syntax includes commands for
both the Oneway ANOVA, and an additional
command to graph the results:
24
Example 8.1 Using SPSS (cont.)
The results appear in the SPSS output viewer:
This ANOVA table simply reproduces the
Omnibus test illustrated previously.
25
Example 8.1 Using SPSS (cont.)


The table above compares all possible mean pairs of
the three conditions.
As indicated above, there are only three unique pairs,
so this table is repetitive in sections
(i.e., Pizza-Beer p-value = Beer-Pizza p-value).
26
Example 8.1 Using SPSS (cont.)
The table
opposite is a
simple
representation
of which
“means” are
significantly
different from
one another.



Condition means which are significantly different
(p < .05) appear in separate columns.
In our example, the means for each condition appear
in separate columns.
27
Example 8.1 Using SPSS (cont.)
The graph
opposite is
simply a
reproduction of
the figure
illustrated
earlier. It can be
modified within
SPSS to
improve its
appearance and
make it
conform to
APA style.
28
The conclusion (again)

Our conclusions can be restated given the exact
probabilities provided in SPSS:
There was model significance for the ANOVA,
F(2,12) = 30.00, p < .05, indicating at least one
significant difference among the means. Scheffé
posthoc comparisons showed that weight gain
was higher in the Cream condition (M = 6.00)
than the Beer condition (M = 4.00), p =.01 (twotailed). In turn, weight gain was lower in the
Pizza condition (M = 2.00) than either the Beer
Condition, p = .01 (two-tailed), or the Cream
condition, p < .01 (two-tailed).
29
Thus concludes 
Scheffé Post-Hoc
Comparisons
Advanced Research Methods in
Psychology
- lecture -
Matthew Rockloff
30