Transcript RELIABILITY AND VALIDITY © LOUIS COHEN, LAWRENCE MANION & KEITH MORRISON STRUCTURE OF THE CHAPTER • • • • • • • • • • Defining validity Validity in quantitative research Validity in qualitative research Types.

RELIABILITY AND VALIDITY
© LOUIS COHEN, LAWRENCE MANION
& KEITH MORRISON
STRUCTURE OF THE CHAPTER
•
•
•
•
•
•
•
•
•
•
Defining validity
Validity in quantitative research
Validity in qualitative research
Types of validity
Triangulation
Validity in mixed methods research
Ensuring validity
Reliability
Reliability in quantitative research
Reliability in qualitative research
STRUCTURE OF THE CHAPTER
•
•
•
•
•
•
Validity and reliability in interviews
Validity and reliability in experiments
Validity and reliability in questionnaires
Validity and reliability in observations
Validity and reliability in tests
Validity and reliability in life histories
BASES OF VALIDITY IN
QUANTITATIVE RESEARCH
Controllability
BASES OF VALIDITY IN
QUALITATIVE RESEARCH
Natural
Isolation, control, manipulation of
Variables
Replicability
Predictability
Thick description
Generalizability
Context-freedom
Fragmentation and atomization
Uniqueness
Context-boundedness
Holism
Randomization of samples
Purposive sample/no sampling
Neutrality
Value-ladenness of observations
Objectivity
Observability
Inference
Confirmability
Observable and non-observable
meanings/ intentions
Description, inference, explanation
‘Etic’ research
Observations
‘Emic’ research
Meanings
Uniqueness
Emergence, unpredictability
BASES OF RELIABILITY IN
QUANTITATIVE RESEARCH
Reliability
BASES OF RELIABILITY IN
QUALITATIVE RESEARCH
Dependability
Demonstrability
Stability and replicability
Parallel forms
Context-freedom
Objectivity
Coverage of domain
Verification of data and analysis
Answering research questions
Meaningfulness to the research
Parsimony
Internal consistency
Generalizability
Inter-rater reliability & triangulation
Accuracy and precision
Neutrality
Consistency
Alternative forms (equivalence)
Trustworthiness
Stability and replicability
Parallel forms
Context-specificity
Authenticity and confirmability
Comprehensiveness of situation
Honesty and candour
Depth of response
Meaningfulness to respondents
Richness
Credibility
Transferability
Inter-rater reliability and triangulation
Accuracy and comprehensiveness
Multiple interests represented
Consistency
VALIDITY IN QUANTITATIVE AND
QUALITATIVE RESEARCH
• Validity in quantitative research often
concerns: objectivity, generalizability,
replicability, predictability, controllability,
nomothetic statements.
• Validity in qualitative research often
concerns: honesty, richness, authenticity,
depth, scope, subjectivity, strength of feeling,
catching uniqueness, idiographic statements.
TYPES OF VALIDITY
•
•
•
•
•
•
•
•
•
•
Catalytic
Concurrent
Consequential
Construct
Content
Criterion-related
Convergent & discriminant
Cross-cultural
Cultural validity
Descriptive
•
•
•
•
•
•
•
•
•
•
Ecological
Evaluative
External
Face
Internal
Interpretive
Jury
Predictive
Systemic
Theoretical
VALIDITY IN QUANTITATIVE RESEARCH
•
•
•
•
•
•
Concurrent
Construct
Content
Criterion-related
Convergent & discriminant
Cross-cultural
•
•
•
•
•
•
•
Evaluative
External
Face
Internal
Jury
Predictive
Theoretical
VALIDITY IN MIXED METHODS
RESEARCH
•
•
Representation
Legitimation
 Sample integration
 Inside-outside
 Weakness minimization
 Sequential
 Conversion
 Paradigmatic mixing
 Commensurability
 Multiple validities
 Political
 Integration (of methods)
HISTORY
MATURATION
TESTING
DIRECTION
OF CAUSALITY
TYPE 1 AND
TYPE 2
ERRORS
INSTRUMENTATION
THREATS TO
VALIDITY AND
RELIABILITY
EXPERIMENTAL
MORTALITY
OPERATIONALIZATION
CONTAMINATION
REACTIVITY
ESTABLISHING VALIDITY IN
QUALITATIVE RESEARCH
•
•
•
•
•
•
•
•
•
•
•
•
Prolonged engagement in the field
Persistent observation
Triangulation
Leaving an audit trail
Respondent validation
Weighting the evidence (giving priority)
Checking for representativeness
Checking for researcher effects
Making contrast/comparisons
Theoretical sampling
Checking the meaning of outliers
Using extreme cases
ESTABLISHING VALIDITY IN
QUALITATIVE RESEARCH
•
•
•
•
•
•
•
•
•
•
•
•
Ruling out spurious relations
Replicating a finding
Referential adequacy
Following up surprises
Structural relationships
Peer debriefing
Rich and thick description
Looking for possible sources of invalidity
Assessing rival explanations
Negative case analysis
Confirmatory data analysis
Effect sizes
THREATS TO VALIDITY IN
QUANTITATIVE RESEARCH
•
•
•
•
•
•
•
•
•
•
History
Maturation
Statistical regression
Testing
Instrumentation
Selection Bias
Experimental mortality
Instrument reactivity
Selection-maturation interaction
Type I and Type II errors
VALIDITY PROBLEMS IN
CROSS-CULTURAL RESEARCH
• Failure to operationalize elements of cultures
• Whose construction of ‘culture’ to adopt: ‘emic’/‘etic’
• False attribution of causality to cultural factors rather than
non-cultural factors
• Directions of causality
• Ecological fallacy
• Sampling and instrumentation
• Convergent and discriminant validity
• Response bias and preparation of participants
• Language problems
• Problems of equivalence (conceptual, psychological,
meaning, instrument, understanding, significance, relevance,
measurement, linguistic)
THREATS TO EXTERNAL VALIDITY
IN QUANTITATIVE RESEARCH
• Failure to describe independent variables explicitly
• Lack of representativeness of available and target
populations
• Hawthorne effect
• Inadequate operationalizing of dependent variables
• Sensitization/reactivity to experimental/research conditions
• Interaction effects of extraneous factors and experimental/
research treatments
• Invalidity or unreliability of instruments
• Ecological validity
• Multiple treatment validity
THE HAWTHORNE EFFECT
•
•
•
•
Between 1927 and 1932 researchers carried out
experiments at the Western Electric Company’s
Hawthorne plant.
Purposes: To examine the effects of changes of
working conditions on output of workers
Sample: Six women, chosen as average workers
Method: Women worked in a test room. Output
measured under different conditions (e.g. no change
→ change to method of payment → introduce two
rest periods → introduce six rest periods →
changes in lighting conditions, early clocking-off,
five-day working week → return to initial conditions
Duration: 15 weeks
THE HAWTHORNE EFFECT
• Results: Output rose steadily during test
period and after the test period.
• Conclusion: Output did not seem to depend
on test conditions. Increased output seemed
to be due to the fact that the people had been
involved in the experiment itself, i.e. the act of
research had affected the results. The
results were a research of the research itself.
• Implications: The act of being involved in
research itself affects the results.
THREATS TO EXTERNAL VALIDITY
IN QUALITATIVE RESEARCH
•
•
•
•
Selection effects
Setting effects
History effects
Construct effects
ENSURING VALIDITY AT THE
DESIGN STAGE
•
•
•
•
•
•
•
•
Choose an appropriate time scale;
Ensure adequate resources for the research
Select appropriate methodology
Select appropriate instruments
Use an appropriate sample
Ensure reliability
Select appropriate foci
Avoid having biased researcher(s)
ENSURING VALIDITY AT THE
DATA COLLECTION STAGE
•
•
•
•
•
•
•
•
•
•
•
Reduce the Hawthorne effect
Minimize reactivity
Avoid drop-out rates amongst respondents
Take steps to avoid non-return of questionnaires
Avoid too long or too short an interval between pre-tests
and post-tests
Ensure inter-rater reliability
Match control and experimental groups
Ensure standardized procedures for gathering data
Build on the motivations of respondents
Tailor instruments to situational factors
Address researcher characteristics
ENSURING VALIDITY AT THE
DATA ANALYSIS STAGE
•
•
•
•
•
•
•
•
•
•
•
•
Use respondent validation;
Avoid subjective interpretation of data
Reduce the halo effect
Use appropriate statistical treatments
Recognize extraneous factors which may affect data
Avoid poor coding of qualitative data
Avoid making inferences/generalizations beyond the data
Avoid equating correlations and causes
Avoid selective use of data
Avoid unfair aggregation of data
Avoid degrading the data;
Avoid Type I and/or Type II errors
ENSURING VALIDITY AT THE
DATA REPORTING STAGE
• Avoid using data selectively and unrepresentatively
• Indicate the context and parameters of the
research
• Present the data without misrepresenting the
message
• Make claims which are sustainable by the data
• Avoid inaccurate or wrong reporting of data
• Ensure that the research questions are answered
• Release research results neither too soon nor too
late
RELIABILITY IN QUANTITATIVE
AND QUALITATIVE RESEARCH
• Reliability in quantitative research:
– consistency (stability), accuracy,
predictability, equivalence, replicability,
concurrence, descriptive and causal
potential.
• Reliability in qualitative research:
– accuracy, fairness, dependability,
comprehensiveness, respondent validation,
‘checkability’, empathy, uniqueness,
explanatory and descriptive potential,
confirmability.
TYPES OF RELIABILITY IN
QUANTITATIVE RESEARCH
• Reliability as stability:
– Consistency over time and samples;
• Reliability as equivalence:
– Equivalent forms of same instrument;
– Inter-rater reliability;
• Reliability as internal consistency:
– Split half reliability (e.g. for test items)
TRIANGULATION
•
•
•
•
•
•
•
•
•
Methodologies
Instruments
Researchers
Time
Location
Theories
Samples
Participants
Data
SPLIT-HALF RELIABILITY
(Spearman-Brown)
Reliability =
2r
1 r
r = the actual correlation between the two halves
of the instrument (e.g. 0.85);
Reliability =
2 ( 0.85 )
1 0.85
=
1.70
185
= 0.919
RELIABILITY IN QUALITATIVE
RESEARCH
•
•
•
•
•
•
•
•
Credibility
Neutrality
Confirmability
Dependability
Consistency
Applicability
Trustworthiness
Transferability
RELIABILITY AND REPLICATION
IN QUALITATIVE RESEARCH
Repeat:
•
•
•
•
•
The status position of the researcher
The choice of informants/respondents
The social situations and conditions
The analytic constructs used
The methods of data collection and analysis
Address:
•
•
•
•
Stability of observations
Parallel forms
Inter-rater reliability
Respondent validation
IMPROVING RELIABILITY
•
•
•
•
•
Minimise external sources of variation;
Standardise conditions under which
measurement occurs;
Improve researcher consistency;
Broaden the sample of measurement
questions by:
a) adding similar questions to the
instrument;
b) increasing the number of researchers
(triangulation);
c) increasing the number of occasions in
an observational study.
Exclude extreme responses (outliers).
RELIABILITY AND VALIDITY AT
ALL STAGES
•
•
•
•
•
•
•
Design and methodology
Sampling
Instrumentation
Timing
Data collection
Data analysis
Data reporting