Transcript Testing Theories: The Problem of Sampling Error The problem of sampling error • It is often the case—especially when making point predictions—that what.

Testing Theories: The Problem of
Sampling Error
The problem of sampling error
• It is often the case—especially when making point
predictions—that what we observe differs from what
our theory predicts.
• There are at least three reasons why we may observe
something other than what our theory predicts.
– the theory is incomplete or dead wrong
– the data are imprecise
– sampling error
• Today we’re going to spend some time discussing the
problem of sampling error, and some of the ways in
which psychologists deal with it in their research.
Sampling error
• Sampling error refers to the discrepancy between
sample statistics and population parameters.
• Sampling error occurs in research because we’re
working with a subset of the population of interest.
– Sometimes the population is sociological (e.g., US citizens
registered to vote); sometimes it is an abstract-mathematical value
(e.g., p(heads) = .5 for a coin flip).
• In either case, it is necessary to know how much
sampling error may be present in our data, and to
take those errors into account when empirically
evaluating our theories.
The problem
• Why is sampling error a problem in psychological
research?
The problem
• Why is sampling error a problem in psychological
research?
• Imagine the following research scenario
– We are interested in the extra-sensory perception
(ESP), and we are trying to determine whether
people can correctly guess which of four cards
someone is thinking about.
– Our sample of 23 people gets 28% of their
guesses correct.
– Is this evidence for extra-sensory abilities?
The problem
• It is important to note that, even if people lack ESP
and are simply guessing, sometimes they’ll be
correct. In fact, we expect them to be correct 25% of
the time, on average.
• More importantly, sometimes people will do better
than 25%--even if they are simply guessing.
The problem
• How often will this occur? What distribution of correct
guesses should we expect if, in fact, the true
probability (population parameter) of a correct guess
is 25% (p = .25) and we test 23 people?
• Binomial theorem: gives us the expected number of
“hits” (a sample value) for a given number of trials
and a given probability (a population value) for
getting a hit.
0.20
0.10
0.05
0.0
Probability
0.15
sampling
distribution
0
0.04 0.09 0.13 0.17 0.22 0.26 0.3 0.35 0.39 0.43 0.48 0.52 0.57 0.61
Perc entage c orrec t
How likely are the various outcomes?
• In this case we see that it would not be unusual for
people to be right 30% of the time, even if they were
simply guessing.
• In fact, in this situation, approximately 95% of the
time we would expect people to be correct between
4% and 43% of the time.
Dealing with sampling error
• How can we deal with the problem of sampling error
in psychological research?
• The most straight-forward way is to minimize it by
using large sample sizes.
In this scenario, the sample size
is 200. Notice that there is
much less of variation in the
sample correlations.
3
2
D ensity
1.0
1
0.5
0
0.0
D ensity
4
1.5
5
In this scenario, the sample size
is 20. Notice that there is a lot
of variation in the sample
correlations.
-1.0
-0.5
0.0
0.5
1.0
-1.0
-0.5
C orrelation
0.0
0.5
1.0
C orrelation
population r = .20
~95% fall between -.25 and .60
~95% fall between .06 and .33
Sampling distributions and point
predictions
• It is often necessary to take into account sampling
distributions when testing point predictions.
• For example, if our theory predicts that the correlation
between television viewing and obesity should be
zero when we hold exercise constant, we know that
we might not observe a correlation of exactly zero
due to sampling error--even if our theory is correct.
Sampling distributions and point
predictions
• By constructing a sampling distribution, we can
determine what range of correlations we’re likely to
observe.
– If the correlation we observe in our study is within
the range that we might expect given sampling
error, then we might not count this as a problem
for the theory.
– If, however, the correlation is much bigger than
what we might expect given sampling error, we
might count it against the theory.
2
1
0
Density
3
If we were to observe a
correlation of .30, we may begin
to question our theory since, if
our theory were true, we would
expect to observe correlations
falling roughly between -.20 and
.20.
-0.4
-0.2
0.0
Correlation
0.2
0.4
Sampling distributions and directional
predictions
• When our theory makes a directional prediction, it is
very difficult to generate the appropriate sampling
distribution.
– If ESP works, the population value of p could be
anywhere between .26 and 1.00.
• As a result, when a directional prediction is made,
researchers often test what is sometimes called the
null hypothesis—the hypothesis that there is no
association between variables (r = 0, mean difference
= 0, p = .25, etc.)
Significance tests
• Basic logic of a null-hypothesis significance test:
– The theoretically derived hypothesis (people have
ESP abilities) is that p > .25. (directional)
– The null hypothesis (people don’t have ESP
abilities) is that p = .25.
– These two hypotheses are mutually exclusive.
– If we can show that the proportion correct that we
observe in our study is unlikely if the null
hypothesis is true, then that indirectly lends
credence to the theoretical hypothesis.
Additional example
• We assume that a certain drug may improve people’s
memory
• We randomly assign people to a drug condition or a
placebo condition
– theoretical hypothesis (directional): the drug group will have
better memory performance than the other group (Mdrug –
Mno drug > 0)
– null hypothesis (point prediction): the drug group will have
the same performance as the control group (Mdrug – Mno drug =
0)
– If we can show that the observed difference is too large to be
explained by the null hypothesis, then that may be taken as
indirect support for our theoretical hypothesis.
Significance tests
• Some commonly used null-hypothesis significance
tests
– analysis of variance (ANOVA)
– t-tests
– z-tests
– chi-squares
• The use of null-hypothesis significance tests is very
common in psychology, but hotly debated.
Summary
• Any time we draw a sample from a population, there
is likely to be some degree of sampling error present.
• Sampling error can interfere with our ability to test a
theory because it will lead the values we observe to
deviate from those expected even if the theory is
true.
• One way to handle sampling error is to minimize it by
using large sample sizes.
• When researchers cannot make point predictions,
they often use significance tests and test the null
hypothesis instead.