Transcript Assessment of QR

A Quantitative Literacy
Assessment Rubric
Development & Lessons Learned
- Stuart Boersma, Central Washington Univ.
- Caren Diefenderfer, Hollins University
- Shannon Dingman, Univ. of Arkansas
- Bernie Madison, Univ. of Arkansas
Supported by the
National Science Foundation
DUE-0715039
Numeracy Article: Quantitative Reasoning in the
Contemporary World 3: Assessing Student
Learning
http://services.bepress.com/numeracy/
Our Course
• Uses case studies centered on one or more media
articles
• Each case study contains study questions focusing
on the quantitative content of the article
• Student responses often involve critiquing,
analyzing, decision making, and explaining.

Making Assessment Difficult!!
QRCW Casebook: Case Studies for Quantitative Reasoning: A Casebook of Media
Articles, 2nd edition, Pearson Custom Publishing. ISBN13: 978-0-558-19880-0.
Need: Instrument that we could apply to these
non-standard student responses.
A Little History
AAC&U VALUE Project:
•Developed for institutional level assessment (not
for grading student work)
•15 areas, 10 of which are termed Intellectual and
Practical skills:
•Inquiry and analysis, critical thinking, written
communication, oral communication, reading,
quantitative literacy, information literacy,
teamwork, problem solving
Interpretation
Representation
Calculation
Application/Analysis
Assumptions
Communication
Alignment of QL VALUE Rubric w/
QRCW course
Missing Competencies?:
– Critical Reading
– Number sense
– Synthesis
Folded into “Interpretation”
Changed “application/analysis”
to “analysis/synthesis”
Scale
– No “zero”
– “4” designated capstone
achievement; not a
consideration for student
work in a QL course
New Scale: 0-3
Achievement Level
Quantitative Literacy Core
Competency
Interpretation
Ability to glean and explain
mathematical information presented
in various forms (e.g. equations,
graphs, diagrams, tables, words)
Representation
Ability to convert information from
one mathematical form (e.g.
equations, graphs, diagrams, tables,
words) into another.
Calculation
Ability to perform arithmetical and
mathematical calculations.
3
2
1
0
Correctly identifies all
relevant information.
Correctly identifies some,
but not all, relevant
information.
Some relevant information
is identified, but none is
correct.
No relevant information
identified.
All relevant conversions are
present and correct.
Some correct and relevant
conversions are present but
some conversions are
incorrect or not present.
Some information is
converted, but it is irrelevant
or incorrect.
No conversion is attempted.
Calculations related to the
problem are correct and lead
to a successful completion
of the problem.
Calculations related to the
problem are attempted but
either contain errors or are
not complete enough to
solve the problem.
Calculations related to the
problem are attempted but
contain errors and are not
complete enough to solve
the problem.
Calculations given are not
related to the problem, or no
work is present.
Analysis/Synthesis
Ability to make and draw
conclusions based on quantitative
analysis.
Uses correct and complete
quantitative analysis to
make relevant and correct
conclusions.
Assumptions
Ability to make and evaluate
important assumptions in
estimation, modeling, and data
analysis.
All assumptions needed are
present and justified when
necessary.
Communication
Ability to explain thoughts and
processes in terms of what
evidence is used, how it is
organized, presented, and
A correct and complete
explanation is clearly
presented.
Quantitative analysis is
given to support a relevant
conclusion but it is either
An incorrect quantitative
only partially correct or
analysis is given to support a
partially complete (e.g. there
conclusion.
are logical errors or
unsubstantiated claims).
At least one correct and
relevant assumption is given
Attempts to describe
(perhaps coupled with
assumptions, but none of the
erroneous assumptions), yet
assumptions described are
some important assumptions
relevant.
are not present.
A partially correct relevant
explanation is present, but
incomplete or poorly
presented.
A relevant explanation is
present, but is illogical,
incorrect, illegible, or
incoherent.
Either no reasonable
conclusion is made or, if
present, is not based on
quantitative analysis.
No assumptions present.
No relevant explanation is
provided.
Use of QL VALUE Rubric in
QRCW course
Descriptions of achievement levels not
objective enough for our uses:
Uses
quantitative
Skillfully
convertsanalysis
of
data asinformation
the basis for
relevant
competent
judgments,
into insightful
drawing
reasonable
mathematical
conclusions…
portrayal…
3
4
v
Competently
Uses
quantitative
converts
analysis
of data asinformation
relevant
the basis for
workmanlike
into
an appropriate
judgments,
and
drawingmathematical
desired
plausible
conclusions…
portrayal
3
2
Achievement Level
Quantitative Literacy Core
Competency
Interpretation
Ability to glean and explain
mathematical information presented
in various forms (e.g. equations,
graphs, diagrams, tables, words)
Representation
Ability to convert information from
one mathematical form (e.g.
equations, graphs, diagrams, tables,
words) into another.
Calculation
Ability to perform arithmetical and
mathematical calculations.
3
2
1
0
Correctly identifies all
relevant information.
Correctly identifies some,
but not all, relevant
information.
Some relevant information
is identified, but none is
correct.
No relevant information
identified.
All relevant conversions are
present and correct.
Some correct and relevant
conversions are present but
some conversions are
incorrect or not present.
Some information is
converted, but it is irrelevant
or incorrect.
No conversion is attempted.
Calculations related to the
problem are correct and lead
to a successful completion
of the problem.
Calculations related to the
problem are attempted but
either contain errors or are
not complete enough to
solve the problem.
Calculations related to the
problem are attempted but
contain errors and are not
complete enough to solve
the problem.
Calculations given are not
related to the problem, or no
work is present.
Analysis/Synthesis
Ability to make and draw
conclusions based on quantitative
analysis.
Uses correct and complete
quantitative analysis to
make relevant and correct
conclusions.
Assumptions
Ability to make and evaluate
important assumptions in
estimation, modeling, and data
analysis.
All assumptions needed are
present and justified when
necessary.
Communication
Ability to explain thoughts and
processes in terms of what
evidence is used, how it is
organized, presented, and
contextualized.
A correct and complete
explanation is clearly
presented.
Quantitative analysis is
given to support a relevant
conclusion but it is either
An incorrect quantitative
only partially correct or
analysis is given to support a
partially complete (e.g. there
conclusion.
are logical errors or
unsubstantiated claims).
At least one correct and
relevant assumption is given
Attempts to describe
(perhaps coupled with
assumptions, but none of the
erroneous assumptions), yet
assumptions described are
some important assumptions
relevant.
are not present.
A partially correct relevant
explanation is present, but
incomplete or poorly
presented.
A relevant explanation is
present, but is illogical,
incorrect, illegible, or
incoherent.
Either no reasonable
conclusion is made or, if
present, is not based on
quantitative analysis.
No assumptions present.
No relevant explanation is
provided.
Frequency and Prevalence of Core Competencies in QRCW Casebook
Number of case
Number of case
studies which
studies (N=24) which
have 50% or
have at least one
more questions
question mapped to
mapped to
competency
competency
Competency
Number
of
questions
Percent
of
questions
Interpretation
152
65%
24
18
68
29%
19
6
Calculation
101
43%
21
9
Analysis/
Synthesis
79
34%
20
7
Assumptions
14
6%
10
0
Communication
53
23%
18
4
Representation
Reliability of QLAR
• Focus on Case Study 4.3
• Readers A and B scored questions 3a, 3b, 3c,
and 3d on 24 student papers (section 1)
• Readers C and D scored 2b, 2c, 2d, 3a, and 3d
from 24 papers (section 2)
Core Competency for Study Questions
Question
2b
2c
2d
3a
3b
3c
3d
Competencies
Assumptions
Assumptions
Assumptions
Representation, calculation,
analysis/synthesis, communication
Interpretation, analysis/synthesis
Interpretation, analysis/synthesis
Representation, calculation
Reliability Test 1: Readers A and B
Disagreement Level Per Core Competency
QL Competency
Interpretation
Representation
Calculation
Analysis/
Synthesis
Communication
# of
disagreements
1
1
3
2
0
N
% disagreement
48
48
46
72
2.08%
2.08%
6.25%
2.78%
24
0%
Overall:
•Readers A and B disagreed on 7/238 = 2.94% of
assessments.
•Average of 7 differences: 1.23
Reliability Test 2: Readers C and D
Disagreement Level Per Core Competency
QL Competency
# of
disagreements
2
N
% disagreement
42
4.76%
Representation
2
28
10.71%
Calculation
6
28
21.43%
Analysis/
Synthesis
Communication
0
14
0%
5
14
35.7%
Assumptions
Overall:
•Readers C and D disagreed on 15/126 = 11.9% of
assessments.
•Average of 7 differences: (just over) 1
Added Value of QLAR
1. Allows one to align student prompts with an
organized thought process as represented by
the rubric
To solve a complex problem a student needs to read and
glean information, convert the information into a usable
form, calculate, make or draw a conclusion, and then
communicate the result via explanation.
Scaffold student learning by sorting case
Students often do not recognize these as components of
studiesresponse.
by competency:
with through mastery of
a strong
GuidingStart
students
Interpretation
& Representation
and build up
these
competencies
is crucial!
to more complex prompts.
Added Value of QLAR
2. Improved student prompts.
Find out how the
Describe how the
Be &sure to differentiate between
Standard
Standard & Poor’s
information
takenindex is
Poor’s quantitative
500 stock
500 stock
index isdirectly from the article
and extra
computed.
Use a quantitative argument
computed.
assumptions
you question]
made.
to…[answer some
Justify any extra assumptions.
Explain your reasoning!
QLAR
Quantitative Literacy Core
Competency
Interpretation
Ability to glean and explain
mathematical information presented
in various forms (e.g. equations,
graphs, diagrams, tables, words)
Representation
Ability to convert information from
one mathematical form (e.g.
equations, graphs, diagrams, tables,
words) into another.
Calculation
Ability to perform arithmetical and
mathematical calculations.
Achievement Level
3
2
1
0
Correctly identifies all
relevant information.
Correctly identifies some,
but not all, relevant
information.
Some relevant information
is identified, but none is
correct.
No relevant information
identified.
All relevant conversions are
present and correct.
Some correct and relevant
conversions are present but
some conversions are
incorrect or not present.
Some information is
converted, but it is irrelevant
or incorrect.
No conversion is attempted.
Calculations related to the
problem are correct and lead
to a successful completion
of the problem.
Calculations related to the
problem are attempted but
either contain errors or are
not complete enough to
solve the problem.
Calculations related to the
problem are attempted but
contain errors and are not
complete enough to solve
the problem.
Calculations given are not
related to the problem, or no
work is present.
Analysis/Synthesis
Ability to make and draw
conclusions based on quantitative
analysis.
Uses correct and complete
quantitative analysis to
make relevant and correct
conclusions.
Assumptions
Ability to make and evaluate
important assumptions in
estimation, modeling, and data
analysis.
All assumptions needed are
present and justified when
necessary.
Communication
Ability to explain thoughts and
processes in terms of what
evidence is used, how it is
organized, presented, and
contextualized.
A correct and complete
explanation is clearly
presented.
Quantitative analysis is
given to support a relevant
conclusion but it is either
An incorrect quantitative
only partially correct or
analysis is given to support a
partially complete (e.g. there
conclusion.
are logical errors or
unsubstantiated claims).
At least one correct and
relevant assumption is given
Attempts to describe
(perhaps coupled with
assumptions, but none of the
erroneous assumptions), yet
assumptions described are
some important assumptions
relevant.
are not present.
A partially correct relevant
explanation is present, but
incomplete or poorly
presented.
A relevant explanation is
present, but is illogical,
incorrect, illegible, or
incoherent.
Either no reasonable
conclusion is made or, if
present, is not based on
quantitative analysis.
No assumptions present.
No relevant explanation is
provided.
Interpretation
Ability to glean and explain
mathematical information
presented in various forms
(e.g. equations, graphs,
diagrams, tables, words)
2
Correctly
identifies some,
but not all,
relevant
information.
3
Correctly identifies all
relevant information.
1
Some relevant
information is
identified, but none
is correct.
0
No relevant
information
identified.
Representation
Ability to convert
information from one
mathematical form (e.g.
equations, graphs,
diagrams, tables, words)
into another.
2
Some correct and
relevant
conversions are
present but some
conversions are
incorrect or not
present.
3
All relevant
conversions are
present and correct.
1
Some information
is converted, but it
is irrelevant or
incorrect.
0
No conversion
is attempted
Calculation
3
Ability to perform
arithmetical and
mathematical calculations.
Calculations related to the problem
are correct and lead to a successful
completion of the problem.
2
1
0
Calculations related
to the problem are
attempted but either
contain errors or are
not complete enough
to solve the
problem.
Calculations related to
the problem are
attempted but contain
errors and are not
complete enough to
solve the problem.
Calculations given
are not related to
the problem, or no
work is present.
3
Analysis/Synthesis
Ability to make and draw
conclusions based on
quantitative analysis.
Uses correct and complete
quantitative analysis to make
relevant and correct conclusions.
2
Quantitative
analysis is given to
support a relevant
conclusion but it is
either only partially
correct or partially
complete (e.g. there
are logical errors or
unsubstantiated
claims).
1
0
An incorrect
quantitative analysis is
given to support a
conclusion.
Either no
reasonable
conclusion is made
or, if present, is not
based on
quantitative
analysis.
Assumptions
3
Ability to make and
evaluate important
assumptions in estimation,
modeling, and data
analysis.
All assumptions needed are present
and justified when necessary.
2
At least one correct
and relevant
assumption is given
(perhaps coupled
with erroneous
assumptions), yet
some important
assumptions are not
present.
1
0
Attempts to describe
assumptions, but none
of the assumptions
described are relevant.
No assumptions
present.
Communication
3
Ability to explain thoughts
A correct and complete explanation
is clearly presented.
and processes in terms of
what evidence is used, how
it is organized, presented,
and contextualized.
2
1
0
A partially correct
relevant explanation
is present, but
incomplete or poorly
presented.
A relevant explanation
is present, but is
illogical, incorrect,
illegible, or
incoherent.
No relevant
explanation is
provided.