Transcript Reliability and Validity

Reliability and Validity
Quality of Data
1
Are we testing what we think
we’re testing?
2
Quantitative Data


Reliability
Validity



Face
External
Internal
3
Reliability


Implies that the same data would have been
collected each time over repeated tests/
observations.
Would a particular technique (or survey
question) yield the same result each time?


“Did you go to church last week?” vs. “How
many times have you been to church in your
life?”
Reliability does not ensure accuracy.

Taken from Babbie, E.
4
Reliability




Problem if interpret questions
differently
Poorly worded questions
Inconsistent coding: coding errors as
with open-ended questions
Lack of definition of key terms
5
Reliability


Poorly worded: Does the library have
adequate facilities and equipment for
physically disabled students
Better: Can patrons in wheelchairs
retrieve books from the browsing
collection?
6
Reliability (indicators)





Pretest
Repeat question(s)
Test/retest
Split half and Parallel
Interscore or scorer
7
Validity



A term to describe a measure that
accurately reflects the concept it is intended
to measure.
Which is a more “valid” indicator of
intelligence- an IQ score, or number of
hours spent studying?
Ultimate validity cannot be proven, but can
be supported by face, internal, and external
measures.

Babbie, E.
8
Types of Validity

Face validity: The quality of an
indicator/ question/ test that makes it a
reasonable measure of a variable.


Church attendance is an indication of
religiosity.
Number of grievances filed is an indicator
of worker morale
9
Internal Validity





Approximate truth about inferences regarding
causal relationships
Typically applied to studies using inferential
statistics (i.e. quantitative measures) than
descriptive or observation studies.
Especially useful for studies assessing affects of
programs
Only applicable to the study in question- not
generalizable. Why not?
Key question: Whether observed changes can be
attributed to your program (the cause) and NOT
other possible impacts/ causes.
10
Internal Validity
Trochim, W.
11
Internal Validity




History or specific events: raises the issue that some variable other than
the independent variable accounted for the change in the dependent variable.
E.G.: the length of time between conducting the pretest and posttest may
have a detrimental effect.
Maturation: the change results from biological or psychological processes,
which occurred over time, and not from the treatment itself. Maturation
becomes more a concern the longer the period between the pretest and
posttest
Pretesting: may affect the dependent variable. Pretesting may alert
participants or educate them about the topic under investigation. Therefore if
subjects are administered a posttest, their performance may reflect a marked
improvement
Measuring instruments or observational techniques: These—not the
treatment—may account for the change in the dependent variable. Further,
the validity of study findings may have been influenced by the fact that the
evaluators as observers, raters, graders, interviewers, and coders gained
experience, became tired, obtained a more complete understanding of the
project, or eased their expectations of test subjects
12
Internal Validity (continued)


A nonrandom assignment of subjects to groups
may signify that the groups were dissimilar from the
beginning. Therefore any change might be
attributed to the differential selection of subjects,
rather than the actual treatment.
Statistical regression refers to the tendency for
extreme scores to regress or move toward the
common mean of subsequent measures. The
assignment of subjects to a particular test group on
the basis of extreme views may affect study
findings.
13
Internal Validity (continued)


Mortality refers to the possibility that some subjects
may have dropped out of the study after completion
of the pretest but before the administration of the
posttest. In such instances, every effort should be
made to identify any common patterns or
characteristics to ensure that any difference
between a group’s pretest and posttest scores
cannot be attributed to the loss of subjects.
Interaction refers to the fact that more than one of
the previous threats might be in play. This is
especially likely in those cases where subjects were
not randomly assigned to groups and the evaluation
was based on existing, intact groups.
14
External Validity


The approximate truth of
generalizations drawn from a study.
The degree to which conclusions
drawn from your study sample would
hold true to other persons in other
places at other times

Trochim, W.
15
External Validity
Trochim, W.
16
External Validity



Example: institutions of higher
education in Massachusetts: control,
highest degree offered, and some
characteristics of library (staff number,
budget, and volume number)
Return rate?
Do respondents differ from nonrespondents as a group?
17
Validity

Content validity (for achievement test):
How well does the test sample what
the students learned? How well does a
standardized test cover what was
taught in the information literacy
program?
18
Validity (continued)


Criterion-related (predictive) (attitude
test to predict performance in a library
skills program): Who well does the test
predict achievement for college
freshmen?
Criterion-related (diagnostic): How well
does the test diagnose current
problems with library use?
19
Validity (continued)

Construct validity: How well does the
test measure comprehension of library
use? Does a test on the use of an
OPAC really measure effective and
efficient use rather than one’s ability to
read test items?
20
Qualitative Study Equivalent




Credibility
Dependability
Confirmability
Transferrability
21
Qualitative Reliability

Researcher is the “instrument”- how to test for
reliability?








Provide details of method, and abundance of evidence
Provide evidence of qualifications as observer
Make assumptions (and possible biases) clear
State research questions clearly
Use early stages of study to generate focus
Observe for an adequate period of time, across a full
range of activities
Collect data from multiple sources
Save data for reanalysis
22
Qualitative Validity

Depends upon reliablity. Like reliability, asserted by
documenting steps








Triangulation- data from different sources/ methods
Full documentation of data- “chain of evidence”
Logical connections between data and conclusions
Conscious and deliberate inclusion of data that might not
support thesis
Preparedness to entertain alternatives
Self-reflection, acknowledgement of own biases
Review of preliminary reports by objective observers
Awareness of limitations

Gorman and Clayton
23
Qualitative Study: Increasing
Reliability and Validty
Inquiry
affected by
Results in
Account for
by:
During
After
To lead to
For findings
that are:
Factor
patternings
Noninterpretability
Prolonged
engagement
Persistent
observation
Peer Debriefing
Triangulation
Member checks
Establish
structural
corroboration
(coherence)
Credibility
Plausible
Situational
Uniqueness
Noncomparability
Collect thick
descriptive data
Do theoretical/
positive
sampling
Develop thick
description
Transferability
Context
relevant
-Gorman and Clayton
24
Qualitative Study: Increasing
Reliability and Validity cont’d
Inquiry
affected by
Results in
Account for
by:
During
After
To lead to
For findings
that are:
Instrumental
changes
Instability
Use overlap
methods
Use stepwise
replication
Leave audit trail
Do
dependability
audit (process)
Dependability
Stable
Investigator
Predilections
Bias
Do triangulation
Practice
reflexivity (audit
trail)
Do
confirmability
audit (product)
Confirmability
Investigatorfree
-Gorman and Clayton
25
Example
For a sweeping study
When conduct it?
For how long?
How deal with
reliability and
validity?
Course evaluation
26
References




Babbie, E. (2005). The basics of social research.
Belmont, CA: Wadsworth Publishing.
Gorman, G.E. & Clayton, P. (2005). Qualitative
research for the information professional: A practical
handbook. London: Facet Publishing.
Trochim, W. M. K. (2006). Research methods
knowledge base. Retrieved July 8, 2008 from
http://www.socialresearchmethods.net/kb/intval.php
http://www.socialresearchmethods.net/kb/external.p
hp
27