Transcript Reliability - Youngstown State University

Reliability
As my grand pappy, Old Reliable,
used to say . . .
Who is this famous bloodhound?
What was he noted for saying?
1
What CU?
2
Reliability Topics:
 The Basic Notion of Reliability
 Factors Affecting Reliability
 Methods of Determining Reliability

Methods Used by Professional Test Makers

Method Suggested for Your Own Tests
 Standard Error of Measurement
 Confidence Bands
3
Basic Notions of Reliability
Reliability refers to the reliability of a test score or set of test scores,
not the reliability of the test.
Reliability questions ask: “Are the scores consistent?” “Are they
stable?”
Reliability is a matter of degree; it is NOT all-or-none.
Reliability is not the same as validity – validity asks “Does a test
measure what is suppose to?” (reliability is necessary for, but
not a sufficient condition for, validity) .
Reliability deals with unsystematic error in assessment.
Systematic error (examples, “I test well because I am ‘testwise’” or “I do not test well because English is not my first
language”) will not be uncovered through tests of reliability
4
Factors Affecting Reliability:
Sources of Unreliability
Test Scoring –
difference between two scorers judgments
one scorer over time (fatigue) and/or halo effect
Test Content –
the sample of test items is too small
the sample of test items is not evenly selected across material
Test Administration –
noise, time limits not consistent, physical conditions
Personal Conditions –
temporary ups and downs
(chronic test anxiety would be a systematic error and thus undetectable
through measures of reliability)
Note: None of these factors automatically result in unreliability, but as we build our
assessments, we hope to reduce the impact of these factors. The extent to
which these factors may be affecting test scores is an empirical question and we
can and will address this as we continue.
5
A Bit of Theory (True/Observed)
 The “perfect test” would be unaffected by the sources of unreliability
and on this perfect test each examinee should get his or her true score.
Unfortunately, we know the observed score we get was likely affected
by one or more of the sources of unreliability.
 So, our observed score is likely too high or to low. The difference
between the observed score and the true score we call the error score;
and this score can be positive or negative.
 We can express this mathematically as:
 True Score = Obtained Score +/- Error
 T = O +/- E (or, looking at it another way, O = T +/- E)
 Theory Time: If we could re-administer a test to one person an
infinite number of times, what would expect the distribution of
their scores to look like? Answer: The Bell Shaped Curve. We
will return to this concept when we discuss the standard error of
measurement.
6
Determining Reliability by Using
the Concept of Correlation
 I can use my understanding of correlation (how two things are related)
to come up with a mathematical calculation that will suggest the
strength (or lack of strength) regarding one or more of the sources of
unreliability that I have identified.
 I will be calculating what will be called the reliability coefficient (since it
is a correlation coefficient measuring a type of reliability). This value
will range -1 to +1.
 For example, let’s consider rater reliability. That is, do different scorers
rate equally; or, another concern, does one scorer rate differently over
time. We express that as either



Inter-rater : reliability among raters (international – many nations)
Intra-rater : same rater (intramural sports – within 1 school)
 Note: the hyphen after inter- and intra- may not be used by some
authors
Compute using Spearman Rank Correlation
7
Re-enter the Correlation Coefficient - the calculated number that best
describes the relationship between two variables, but now we will call it the reliability
coefficient
 Reliability coefficient – symbol is “r” – linear relationships
 Range -1.00 through .00 to +1.00
Sign indicates direction
 + indicates that as one variable increases, the other variable
increases

- indicates that as one variable increases, the other variable
decreases
 Number indicates strength
 Although the following table is somewhat arbitrary, the following
thinking might be useful in interpretation:
 -1.0 to -0.7 strong converse association.
 -0.7 to -0.3 weak converse association.
 -0.3 to +0.3 little or no association.
 +0.3 to +0.7 weak direct association.
 +0.7 to +1.0 strong direct association.

8
Some History . . .
Karl Pearson (1857-1936)
 Pearson was a Galton protégé and was appointed
the first Galton Professor or Eugenics (1911) at
University College of London .
 Introduced a new science: "Biometrics" which
integrated statistics with evolutionary theory.
 Advocated social imperialism — "superior" races
and countries should produce more offspring than
those considered to be less developed.
In the United States, Indiana was the first to pass a
pioneering statute (1907) allowing state officials to
sterilize those deemed unfit to breed. California
enacted an even stricter eugenics law. California
made it legal for state officials to asexualize those
considered feeble-minded, prisoners exhibiting sexual
or moral perversions, and anyone with more than
three criminal convictions.
9
More Reliability Approaches to Consider
 Test-retest – (impractical for you; important in standardized
tests)
 Alternate Forms (again, impractical for you but important in
standardized tests)
 Internal Consistency (not appropriate for speeded tests)



Kuder-Richardson (really a series of formulas based on
dichotomously scored items)
Coefficient alpha - Cronbach’s (most widely used as can be
used with continuous item types)
Split-half; odd-even – w/Spearman-Brown correction to apply
to full test (easiest for you to do and understand)
10
Reliability of Your Classroom Tests
 I would recommend doing Split-Half Reliability.
 Step 1 – Split your test into two parts (odd – even).
 Step 2 – Use “Pearson Product Moment Correlation -
Ungrouped Data” to determine “rxy” (rxy represents the
correlation between the two halves of the scale). By doing the
split-half we reduce the number of items which we know will
automatically reduce the reliability, SO
 Step 3 – To estimate reliability of whole test then use the
Spearman Brown “correction” formula
rsb = 2rxy /(1+rxy)
where rsb is the split-half reliability coefficient
11
As a Teacher, What Do I Need to Know
Most About Reliability
 For tests I create myself:
Increasing number of items increases reliability.
 Moderate difficulty level increases reliability.
 Having items measuring similar content increases
reliability.
 For standardized tests I use:


Look for each test’s published reliability data.

Use the published reliability coefficient to
determine the Standard Error of Measurement
(abbreviated SEM) found in the data
 See the following illustration
12
Standard Error of Measurement

The SEM is the standard deviation of a hypothetically infinite number of obtained
scores around a person’s true score.
13
SEM and Confidence Bands
 The SEM is a standard deviation of a distribution assumed to be




normal .
So computing the SEM can help me better interpret scores
Formula: SEM = SD  1 - r
I can take the computed SEM and build a Confidence Band
around my score.
Confidence Band
 68% Confidence Band +/- 1 SEM
 95% Confidence +/- 1.96 SEM
 99% Confidence +/- 2.58 SEM
 I can also do percentiles (a bit harder).
 Many professional test makers give me this information
14
Final Thoughts & Advice
 Use multiple sources of information.
 Find and Use a published test’s SEM to help
interpretation.
 Standard Error of Measurement is distinct from:
 Standard error of mean (samples/populations)
 Standard error of estimate (prediction)
 Reliability for Criterion-referenced Items may use
techniques already covered but sometimes require
special treatment.
 Worry about scorer reliability when score depends
on judgment.
15
More Final Words . . .





Reliability for Sub Scores is problematic since
small clusters are usually quite unreliable.
For important decisions, get reliability >.90.
Be wary of short tests. To increase reliability,
increase number of items, exercises, or
observations.
Occasionally check reliability of your classroom
tests.
Be able to distinguish between reliability and
validity.
16