Transcript Chapter 8 - Graziano & Raulin

Hypothesis Testing, Validity,
and Threats to Validity
Graziano and Raulin
Research Methods: Chapter 8
This multimedia product and its contents are protected under copyright law. The following are prohibited by
law: (1) Any public performance or display, including transmission of any image over a network; (2)
Preparation of any derivative work, including the extraction, in whole or in part, of any images; (3) Any rental,
lease, or lending of the program.
Copyright © Allyn & Bacon (2010)
Testing Hypotheses

Researchers tests specific hypotheses
– Generated from the initial research idea through
a series of steps

A research idea can stimulate dozens of
research hypotheses depending on how
– it is translated into a statement of the problem
– the variables are operationally defined
Copyright © Allyn & Bacon (2010)
The Initial Idea

The initial idea is the starting point
– Often vague or general
– It requires refining before research hypotheses
can be generated

Refinement of the initial idea is based on
(1) a search of relevant research literature
(2) initial observations of the phenomenon

Narrow and formalize the initial idea into a
statement of the problem
Copyright © Allyn & Bacon (2010)
Statement of the Problem

In the form of a question
– Clearly indicates an expected relationship
– Nature of the question dictates the required level
of constraint of a study



Causal questions will require experimental research
Questions about relationships can be answered with
lower-constraint research
Convert into research hypotheses by
operationally defining the variables
Copyright © Allyn & Bacon (2010)
Generating Research
Hypotheses

Ideas lead to
– observations
– library research


Statement of
problem
Problem statements
become research
hypotheses when
constructs are
operationalized
Copyright © Allyn & Bacon (2010)
Operational Definitions



Initially covered in Chapter 3
The procedures used to measure
and/or manipulate variables
Most variables can be operationally
defined in many different ways
Copyright © Allyn & Bacon (2010)
Research Hypotheses



State clearly the expected relationship
between the variables
The form is a declarative statement,
but it is a tentative statement to be
tested in research
Variables in research hypotheses are
stated in operational definition terms
Copyright © Allyn & Bacon (2010)
The Role of Theory

Theory guides all research planning
– Often the primary source of the research
hypothesis
– Guides the selection of variables
– Guides the operational definitions of
variables

Most research is based on multiple,
overlapping, and interacting theories
Copyright © Allyn & Bacon (2010)
Testing Research
Hypotheses

Actually testing three sets of hypotheses
– The null hypothesis
– The confounding variable hypotheses
– The causal hypothesis

Accept causal hypothesis only if you
– reject null hypothesis (statistical analysis)
– rule out each potential confounding variable
hypothesis (based on appropriate controls)
Copyright © Allyn & Bacon (2010)
Evaluating Hypotheses
Copyright © Allyn & Bacon (2010)
Types of Validity
Statistical Validity
 Construct Validity
 External Validity
 Internal Validity

Copyright © Allyn & Bacon (2010)
Statistical Validity


Are the statistical tests accurate?
Threatened by
– Unreliable measures
– Violations of statistical assumptions

Strengthened by
– Using well validated measures
– Having approximately equal sample sizes
in each group (covered in Appendix D)
Copyright © Allyn & Bacon (2010)
Construct Validity


Is our theory the best explanation for the
results?
Threatened by
– Any alternative explanation for the results

Strengthened by
– Using well-validated constructs to build the
theoretical predictions for the study
Copyright © Allyn & Bacon (2010)
External Validity


Do the results apply to the broader
population?
Threatened by
– Unrepresentative samples
– Generalizing beyond the limits of the sample

Strengthened by
– Gathering a representative sample (if possible)
– Clearly describing sample, so that other
researchers will know the limits of generalization
Copyright © Allyn & Bacon (2010)
Internal Validity


Is the independent variable
responsible for the observed changes
in the dependent variable?
Threatened by
– Confounding variables

Strengthened by
– Adding adequate controls to reduce or
eliminate confounding
Copyright © Allyn & Bacon (2010)
Avoiding Confounding

Confounding and internal validity
– Many sources for confounding (covered next)
– With proper controls, confounding can be
virtually eliminated (see Chapter 9)

Confounding and construct validity
– Make sure that you have considered alternative
theoretical explanations for the anticipated
phenomenon
Copyright © Allyn & Bacon (2010)
Confounding Variables 1

Maturation
– Changes due to growth or predictable changes

History
– Changes due to an event that occurs during the
study

Testing
– Changes due to the effects of previous testing
Copyright © Allyn & Bacon (2010)
Confounding Variables 2

Instrumentation
– Any change in the calibration of the measuring
instrument over the course of the study

Regression to the Mean
– Tendency for participants selected because of
extreme scores to be less extreme on a retest

Selection
– Any factor that creates groups that are not equal
at the start of the study
Copyright © Allyn & Bacon (2010)
Confounding Variables 3

Attrition
– Loss of participants during a study; are the
participants who drop out different from those
who continue?

Diffusion of treatment
– Changes in participants’ behavior due to
information they obtained about other conditions

Sequence Effects
– Effects on performance in one condition due to
experience with previous conditions
Copyright © Allyn & Bacon (2010)
Subject Effects

Participants are not passive
– They try to understand the study to help them to
know what they “should do” (termed subject
effects)
– Respond to subtle cues about what is expected
(termed demand characteristics)

Placebo effect: treatment effect due to
expectations that the treatment will work
Copyright © Allyn & Bacon (2010)
Experimenter Effect




Based on the expectations of the researcher
Can affect the outcome of studies if not
controlled
May be due to the experimenter providing
demand characteristics to the participant
Not the same as scientific fraud (which is
deliberate)
Copyright © Allyn & Bacon (2010)
Validity, Control,
and Constraint


Three closely-tied concepts
Validity
– The accuracy of the study or procedure
– Increased by using appropriate control
procedures

The more controls we employ, the
higher the level of constraint
Copyright © Allyn & Bacon (2010)
Ethical Principles

Risk is balanced by reward
– A poorly designed study will provide no useful
information; therefore, any risk would be
unacceptable

Informed Consent
– Virtually guarantees that you will have
confounding due to selection because some
people will refuse to participate
– A small price to pay to maintain ethical
standards
Copyright © Allyn & Bacon (2010)
Summary


Start by building a research hypothesis
Testing the research hypothesis is actually
testing three hypotheses
– (1) null; (2) confounding-variable; (3) causal



Several types of validity
Many potential confounding variables
Subject and experimenter effects can also
affect the outcome of the study
Copyright © Allyn & Bacon (2010)