Transcript Chapter 3

Chapter 5
Reliability and Validity
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Overview
Measuring Variables
 Choosing a Behavior to
Measure
 Overview of Types of
Measurement Errors
Manipulating Variables
 Validity
– Threats to
– Establishing

– Bias
– Random error


Reliability
Validity
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Types of manipulations
Two Types of Measurement
Error


Bias
Random error
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Three “Places” Measurement
Error Can Occur



Observer/Scorer
Participant
Person administering the measure
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Two Types of Observer Error


Observer bias (Scorer bias)
Random observer error
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Minimizing Observer Errors


Why it is more important to reduce
observer bias than random error
Techniques for reducing observer bias*
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Techniques for Reducing
Observer Bias




Eliminating human observer errors by
eliminating the human observer
Limiting human observer errors by
limiting the human observer’s role
Reducing observer bias by making
observers “blind”
Conclusions about reducing observer bias
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Reducing Random Observer
Error

Most of the techniques that reduce
observer bias reduce random observer
error
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Errors in Administering the
Measure

Types
– Experimenter (researcher) bias
– Random error

Solutions
– Blind technique to reduce bias
– Standardization to reduce both bias and
random error
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Errors Due to the Participant


Bias due to the participant (Subject bias)
Random error due to the participant
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Subject (Participant) Bias


Obeying demand characteristics
Social desirability bias
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Conclusions about Reducing
Subject Biases


Blind techniques can reduce demand
characteristics
Making participants anonymous can
reduce social desirability bias
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Summary of Types of
Measurement Error


Try to reduce all forms of measurement
error
Really focus on reducing bias
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Reliability: The (Relative)
Absence of Random Error


The importance of being reliable:
Reliability as a prerequisite to validity
Using test-retest reliability to assess
overall reliability: To what degree is a
measure “random error free”?
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Identifying (and Then Dealing with)
the Main Source of a Measure’s
Reliability Problems



Are observers to blame for low test-retest
reliability?: Assessing observer reliability
Non-observer sources of random error
Using internal consistency measures to
estimate random error due to participants
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Internal Consistency: Test
Questions Should Agree with
Each Other

Random error due to participants may
cause low internal consistency
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Two Solutions to Problems
Caused by Random
Participant Error


Add questions to let random participant
error balance out
Ask better questions to reduce random
participant error
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Measuring Internal
Consistency




Average inter-item correlations as indexes
of internal consistency
Split-half coefficients as indexes of
internal consistency
Additional indexes of internal consistency
Conclusions about internal consistency’s
relationship to reliability
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Conclusions About Reliability



Reliability is a prerequisite for validity
If test-retest reliability is low, try to find
out where reliability problem is and fix it.
Reliability does not guarantee validity
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Beyond Reliability:
Establishing Construct Validity



Content Validity
Internal Consistency
Convergent Validity:
Getting evidence that
you are measuring
the right construct

PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Discriminant
Validity: Showing
that you are not
measuring the wrong
construct
Manipulating Variables




Common threats to a manipulation’s
validity
Evidence used to argue for a
manipulation’s construct validity
Tradeoffs among three common types of
manipulations
Conclusions
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Common Threats to a
Manipulation’s Validity



Random error
Experimenter bias
Subject biases
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Evidence Used to Argue for a
Manipulation’s Construct Validity


Consistency with theory
Manipulation checks
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Tradeoffs Among Three
Common Types of
Manipulations



Instructional manipulations
Environmental manipulations
Manipulations involving stooges
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley
Concluding Remarks
Operational definitions should
 Be consistent with dictionary/theory
definitions
 Be standardized to reduce bias and random
error
 Have evidence to support their validity
PowerPoint presentation to accompany Research Design Explained 6th edition;
©2007 Mark Mitchell & Janina Jolley