Transcript Reliability Lesson Seven

Lesson Seven
Reliability
Contents



Definition of reliability
Indication of reliability: Reliability coefficient
Ways of obtaining reliability coefficient:






Alternate/Parallel forms
Test, retest
Split-half
(Inter-) rater (or scorer) reliability
Two ways of testing reliability
How to make sure the test is reliable
Definition of Reliability


“The consistency of measures across
different times, test forms, raters,
and other characteristics of the
measurement context” (Bachman,
1990, p. 24).
The accuracy or precision with which
a test measures something;
consistency, dependability, or stability
of test results.
Reliability coefficient (r)


To quantify the reliability of a test 
allow us to compare the reliability of
different tests.
0 ≤ r ≤ 1 (ideal r= 1, which means the test gives
precisely the same results for particular testees
regardless of when it happened to be administered).


If r = 1 : 100% reliable
A good achievement test: r>= .90
Alternate/Parallel forms: the most stringent form

Two forms, two
administrations


Equivalent forms (i.e.,
different items testing the
same topic) taken by the
same test taker on
different days
If r is high, this test is said
to have good reliability.
Test plan
Form A
Form B
Test, retest
One
form, two administrations
Test A
Trial 1


Trial 2
The same test is administered to the
same testees with a short time lag, and
then calculate r.
Appropriate for highly speeded test
Split-half (Spearman-Brown Procedure)



One test, one administration
Split the test into halves (i.e., odd
questions vs even questions) to
form two sets of scores.
Internal consistency (also: KR-21, KR-20)
Q1
Q2
Q3
First Half
Q4
Q5
Q6
Second Half
(Inter-) rater (or scorer) reliability



Needed for subjective tests (e.g.,
writing, oral tests) when two or
more independent raters are
involved in scoring.
Raters should be trained before
scoring.
Compare the scores of the same
testee given by different raters. If
r= high, there’s inter-rater reliability.
Ways of testing reliability

Examine the amount of variation



Standard Error of Measurement (SEM)
The smaller the better
Calculate “reliability coefficient”


“r”
The bigger the better
How to make sure the test is reliable
for teachers










Take enough samples of behavior
Try to avoid ambiguous items
Provide clear and explicit instructions
Well layout
Provide uniform and undistracted condition
Try to use objective tests
Try to use direct tests
Have independent, trained raters
Try to identify the test takers by number, not by
names
Try to have more multiple independent scoring in
subjective tests
(Hughes, 1989, pp. 36-41).