Transcript Chapter 3
Chapter 5
Reliability and Validity
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Overview
Measuring Variables
Choosing a Behavior to
Measure
Overview of Types of
Measurement Errors
Manipulating Variables
Validity
– Threats to
– Establishing
– Bias
– Random error
Reliability
Validity
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Types of manipulations
Two Types of Measurement
Error
Bias
Random error
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Three “Places” Measurement
Error Can Occur
Observer/Scorer
Participant
Person administering the measure
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Two Types of Observer Error
Observer bias (Scorer bias)
Random observer error
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Minimizing Observer Errors
Why it is more important to reduce
observer bias than random error
Techniques for reducing observer bias*
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Techniques for Reducing
Observer Bias
Eliminating human observer errors by
eliminating the human observer
Limiting human observer errors by
limiting the human observer’s role
Reducing observer bias by making
observers “blind”
Conclusions about reducing observer bias
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Reducing Random Observer
Error
Most of the techniques that reduce
observer bias reduce random observer
error
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Errors in Administering the
Measure
Types
– Experimenter (researcher) bias
– Random error
Solutions
– Blind technique to reduce bias
– Standardization to reduce both bias and
random error
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Errors Due to the Participant
Bias due to the participant (Subject bias)
Random error due to the participant
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Subject (Participant) Bias
Obeying demand characteristics
Social desirability bias
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Conclusions about Reducing
Subject Biases
Blind techniques can reduce demand
characteristics
Making participants anonymous can
reduce social desirability bias
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Summary of Types of
Measurement Error
Try to reduce all forms of measurement
error
Really focus on reducing bias
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Reliability: The (Relative)
Absence of Random Error
The importance of being reliable:
Reliability as a prerequisite to validity
Using test-retest reliability to assess
overall reliability: To what degree is a
measure “random error free”?
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Identifying (and Then Dealing with)
the Main Source of a Measure’s
Reliability Problems
Are observers to blame for low test-retest
reliability?: Assessing observer reliability
Non-observer sources of random error
Using internal consistency measures to
estimate random error due to participants
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Internal Consistency: Test
Questions Should Agree with
Each Other
Random error due to participants may
cause low internal consistency
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Two Solutions to Problems
Caused by Random
Participant Error
Add questions to let random participant
error balance out
Ask better questions to reduce random
participant error
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Measuring Internal
Consistency
Average inter-item correlations as indexes
of internal consistency
Split-half coefficients as indexes of
internal consistency
Additional indexes of internal consistency
Conclusions about internal consistency’s
relationship to reliability
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Conclusions About Reliability
Reliability is a prerequisite for validity
If test-retest reliability is low, try to find
out where reliability problem is and fix it.
Reliability does not guarantee validity
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Beyond Reliability:
Establishing Construct Validity
Content Validity
Internal Consistency
Convergent Validity:
Getting evidence that
you are measuring
the right construct
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Discriminant
Validity: Showing
that you are not
measuring the wrong
construct
Manipulating Variables
Common threats to a manipulation’s
validity
Evidence used to argue for a
manipulation’s construct validity
Tradeoffs among three common types of
manipulations
Conclusions
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Common Threats to a
Manipulation’s Validity
Random error
Experimenter bias
Subject biases
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Evidence Used to Argue for a
Manipulation’s Construct Validity
Consistency with theory
Manipulation checks
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Tradeoffs Among Three
Common Types of
Manipulations
Instructional manipulations
Environmental manipulations
Manipulations involving stooges
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Concluding Remarks
Operational definitions should
Be consistent with dictionary/theory
definitions
Be standardized to reduce bias and random
error
Have evidence to support their validity
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley