Transcript The Art of Interpreting Test Results

So Much Data, So Little Time:
What Parents and Teachers Need to Know
About Interpreting Test Results
Lee Ann R. Sharman, M.S.
ORBIDA Lecture Series
April 13, 2010
You’re Not Paying Attention!

By the end of our session, you will be
understand these key terms/concepts:
◦ Different strokes for different folks: all tests are
not equal!
◦ Basic statistics you MUST know
 Reliability and validity
 The Bell Curve – a beautiful thing
◦ Error, and why it matters
◦ Common mistakes that lead to poor decisions
 Entitlement





decisions (eligibility)
Skills assessment (diagnostic)
Screening and Progress Monitoring (RtI)
Instructional planning, accommodations and
modifications
Curriculum evaluation – is it working?
Increased focus on data-based decision making to
measure outcomes
Working smarter: ask the right questions!
Understand the differences between types of tests and
what they were designed to measure:
◦
◦
◦
◦
Curriculum-based measures (Dibels, Aimsweb)
Teacher-made criterion referenced tests
Published criterion referenced tests
Norm-referenced tests of Achievement
 OAKS
 Woodcock-Johnson III
◦ Norm-references tests of Cognitive Ability
The test you choose depends on what questions you want to
answer.






School records (file reviews)
Interviews
Medical and Developmental histories
Error analyses
Use of portfolios
Observations
The Snapshot: Point in Time Performance
Measuring Improvement (Change) and
Growth
…You can use a hammer to push in a screw, but a
screwdriver will be easier and more efficient
What OAKS is:
•OAKS is a “Point in Time” measurement,
intended to be used more as a Summative
Assessment. It’s a SNAPSHOT.
•Gives information on group achievement
towards state standards to stakeholders; “Are
enough students in our district meeting
benchmarks?

What OAKS is NOT:
 OAKS is not intended to give information (see
OARs) that will inform instruction or
interventions
 A tool designed for Progress Monitoring
 A measure of aptitude or ability
 A comprehensive measure of identified
content

Response to Intervention – RtI
◦ All models involve tiers of interventions, progress
monitoring, and cut scores to determine who is a
“responder” (or not).
◦ Dibels is a commonly used tool for progress
monitoring


A few different models, but in this case we refer to
measurement of the cognitive abilities underlying
areas of unexpected low academic achievement
Specific cognitive abilities (processing measures,
e.g. Rapid Automatic Naming, Phonemic
Awareness, Long-Term Retrieval) predict reading,
writing, and math acquisition ability
1. WHAT KIND OF TEST IS THIS?
2.
3.
4.
5.
e.g. Norm or Criterion-referenced
What is it used for - the purpose?
Is it valid for the stated purpose (what it measures or
doesn’t measure)?
Is the person administering the test a qualified
administrator?
Are the results valid (test conditions optimal, etc.)








Parental permission…true informed consent
Screening for sensory impairments or
physical problems
File review of school records
Parent/caregiver interview
Documented interventions and quality
instruction
Intellectual and academic assessment
Behavioral assessment or observation
Summary and recommendations
I. Identifying data – the Who, What, When
II. Background Information
A.
B.
C.
D.
E.
Student history
Reason for Referral
Classroom Observation
Parent Information
Instruction received/strategies implemented
Test Results
Test Interpretation
Summary and Conclusions
III.
IV.
V.
A.
B.
C.
Summary
Recommendations for instruction
Recommendations for further assessment, if
needed

Must haves:
◦ Skilled examiner
◦ Optimal test conditions
◦ Cultural bias – be aware
◦ Validity/reliability
◦ Appropriate measures for goal
Kids are more than the scores – the “rule
outs”:
◦ Home/Environmental issues
◦ Sensory acuity problems
◦ Previous educational history
◦ Language factors
 Second language and/or language disorders
◦ Social/Emotional/Behavioral issues


The Matthew Effect – poor reading skills
depress IQ scores (Stanovich)…”The rich get
richer”
The Flynn Effect – IQ is increasing in the
population over time; tests are renormed to
reflect this phenomenon

The devil is in those
details…learn the
basic principles of
Statistics
Simply stated:
Statistics are used to measure things and describe
relationships between things, using numbers
1.
2.
3.
4.
Standard Scores (SS) and Scaled Scores (ss)
Percentile Ranks (% rank)
Age and Grade Equivalents (AE/GE)
Relative Proficiency Index (RPI)
OR, The Normal Frequency Distribution



Mean and standard deviation of the test used
reported
Standard scores, percentile ranks, and
standard errors of measures, with
explanations of each
Both composite or broad scores and subtest
scores, with an explanation of each

Information about developmental ceilings,
functional levels, skill sequences, and
instructional needs upon which
assessment/curriculum linkages can be used
to write the IEP goals

These are raw scores which have been
transformed to have a given mean (average)
and standard deviation (set range or unit of
scores). The student’s test score is compared
to that average. A standard score expresses
how far a student’s score lies above or below
the average of the total distribution of scores.
Composite or
Cluster scores
Standard or scaled
scores
Raw scores


Similar to SS, but in a different form. Allows
us to determine a student’s position (relative
ranking) compared to the standardized
sample
Percentile rank is NOT the same as a percent
score! PR refers to a percentage of persons;
PC refers to a percentage of test items
correct.

Valuable statistic, found only on WJ-III
◦ Written as a percentage, or number out of 90,
indicating percent of proficiency on similar tasks
that students in the comparison group would have
90% success. Correlated with Independent,
Instructional, and Frustration levels (see sample)




Making faulty comparisons: Compare only data sets
measuring the same content, with good content/construct
validity, that are NORMED ON THE SAME POPULATION
Using and AE/GE as a measure of the child’s proficiency/skill
mastery of grade level material
Error exists! Don’t forget about the confidence intervals
◦ SEM creates uncertainty around reporting 1 number
Confusing Percentile RANKS with Percentages:
◦ PR = relative ranking out of 100
◦ Percentage = percentage correct


Age equivalents are developed by figuring out
what the average test score is (the mean) for
a group of children of a certain age taking the
test; not the same as skills
Grade equivalents are developed by figuring
out what the average test score is (the mean)
for a student in each grade.






Commonly used
Misleading
Misunderstood
Difficult to explain
May have little relevance
Avoid in favor of Standard Scores/%Ranks

“When assessed with teacher made tests, Sally
locates information within the text with 60%
accuracy.”
VS.

“Sally’s performance on the OLSAT falls at the
60th %ile rank.”

Are the student’s skills better developed in
one part of a domain than another?
For example:
“While Susan’s Broad Math score was within
the low average range, she performed at a
average level on a subtest that assesses
problem solving, but scored well below
average on a subtest that assesses basic
math calculation.

Test scores don’t support the teacher report
of a weakness?
◦ First, look at differences in task demands of the
testing situation, and in the classroom, when
hypothesizing a reason for the difference.
◦ Look at student’s Proficiency score (RPI) vs.
Standard Score (SS)

“Although weaknesses in mathematics were
noted as a concern by Billy’s teacher, Billy
scored in the average range on assessments
of math skills. These tests required Billy to
perform calculations and to solve word
problems that were read aloud to him. It
was noted he often paused for 10 seconds
or more before starting paper and pencil
tasks in mathematics.”

“Billy’s teacher stated that he does well in
spelling. However, he scored well below
average on a subtest of spelling skills. Billy
appeared to be bored while taking the
spelling test, so a lack of vigilance in his
effort may have depressed his score. Also,
the school spelling tests use words he has
been practicing for a week.

The lower score may indicate that he is
maintaining the correct spelling of words in
long-term memory, and is not able to
correctly encode new words he has not had
time to study.
1. WHAT KIND OF TEST IS THIS?
2.
3.
4.
5.
e.g. Norm or Criterion-referenced
What is it used for - the purpose?
Is it valid for the stated purpose (what it measures or
doesn’t measure)?
Is the person administering the test a qualified
administrator?
Are the results valid (test conditions optimal, etc.)



The good news: We are moving away from
the old “Test and Place” mentality.
The challenge: School teams are using
more comprehensive data sets, which
require more knowledge to interpret
More good news: The best decisions are
made using multiple sources of good
information

“The true utility of assessment is the extent
to which it enables us to find the match
between the student and an intervention that
is effective in getting him or her on track to
reach a meaningful and important goal. The
true validity of any assessment took should
be evaluated by the impact it has on student
outcomes.”
(Cummings/McKenna 2007)

“…one of the problems of writing about
intelligence is how to remind readers often
enough how little an IQ score tells you about
whether or not the human being next to you
is someone whom you will admire or cherish.”
(Herrnstein and Murray)