Transcript How to Norm Rubrics

HOW TO NORM RUBRICS
Ursula Waln, Director of Student Learning Assessment
Central New Mexico Community College
WHAT IS A RUBRIC?
 A rubric is a scoring guide such as:
 A checklist
 A rating scale
 A matrix or list containing descriptions of student work
characteristics at different levels of sophistication, a.k.a. a
descriptive rubric
•
•
Holistic (containing performance levels but not separating out criteria)
Analytic (providing descriptions for each criterion at each
performance level)
Addressed
important
details
Organized
information
logically
Used proper
writing
mechanics
(3)
Proficient
(2)
Developing
Consideration of Diverse Points of View
Developed
key ideas
(1)
 Sanitized hands
 Verified the
patient’s fasting
status
 Asked about
latex sensitivity
 Selected
appropriate
gloves and
tourniquet
 Assembled
necessary
supplies
 Positioned the
patient properly
Descriptive Rubric
Rating Scale
Beginning
Checklist
Beginning
Developing
Proficient
(1)
(2)
(3)
Wholly
dismisses or
disparages
points of view
that diverge
from own
worldview
Identifies
valid
components
of differing
perspectives
but responds
in accordance
with own
worldview
without
reflection
Analyzes the
complexity
and validity
of differing
perspectives
and reevaluates own
perspectives
in light of
alternative
worldviews
 With the exception of checklists, rubrics are used to lend a
level of objectivity to evaluation that is inherently subjective
 Checklists are for use when the demonstration of learning either
is or is not present, with no in-between degrees of manifestation
•
Checklists do not require norming
 Rating scales are the most subjective because they rely on the
scorer’s interpretation of the performance-level headings
 Descriptive rubrics can essentially eliminate subjectivity by
clearly identifying indicators of distinct levels of performance
OBJECTIVE VS. SUBJECTIVE SCORING
And Selecting the Right Tool
WHY NORM A RUBRIC?
 Rubric norming is called for when more than one
person will be scoring students’ work and the results
will be aggregated for assessment of student learning
 To develop shared understanding of the outcome(s) assessed
 To discover any need for editing of the rubric
 To develop scoring consistency among raters
•
•
Minimize the variable of individual expectations regarding rigor
Minimize potential for differences in interpretation of criteria tied
to identification of performance levels
NORMING RUBRICS OR NORMING RATERS?
 Norming rating scales = developing consensus among raters
 What the different performance levels are intended to capture
 What level of rigor should be applied in distinguishing the levels
 All who will be raters should be involved in the norming session(s)
 Norming descriptive rubrics = perfecting the rubric
 The better written the rubric, the less possibility it allows for
differences in interpretation
•
•
Identify and fix gray areas and ambiguities in the rubric
Reduce need for scorers to conform to a group standard for interpretation
WHAT YOU’LL NEED
 A rubric
 A facilitator
 Some work samples
 Student learning outcome statements to which the rubric is tied
 An outline of the steps of the norming process
 Raters (the faculty who will be doing the scoring)
SELECTING WORK SAMPLES
 Real student work or mock-up samples
 If using real student work, redact any identifying information
 Select samples that demonstrate different performance levels
 Plan to have 1 to 3 for each of 2 to 4 scoring sessions
•
Determine number based on the time and complexity of scoring
 If the rubric contains multiple criteria, select samples that display
differing levels of performance on differing criteria (i.e., samples
that are neither all good nor all bad)
OUTLINE OF THE STEPS
I. Orientation
II. Discuss levels and rating criteria/thought processes
III. Score the samples
IV. Compare scores
V. Discuss (and possibly modify the rubric)
VI. Repeat above two steps as needed until consensus is reached
ORIENTING THE RATERS TO THE PROCESS COMPONENTS
 The student learning outcome(s) being assessed
 The purpose of the norming session
 The rubric itself
 How it came to be
 Its intended alignment to the SLO(s)
 Its intended use
 An outline of the process
 Model the thought processes involved in using the rubric
 Entertain perspectives regarding:




The number of performance levels and their headings
The construct validity of the criteria that have been identified
Perceived distinctions between performance levels
Perceptions regarding how the faculty think most of the student
work will be scored and the discriminative value of the rubric
DISCUSSING LEVELS & CRITERIA
Part of the Orientation
SCORING THE SAMPLES
 All raters score the same 1 - 3 samples concurrently
 Raters refrain from discussing the works and/or their scores
during the scoring sessions
 Each time, samples provide a range of skill demonstration
 Start with the most straight-forward samples and work up to
those that require more refined decision-making
COMPARING SCORES
 Look for consistencies and inconsistencies
 Confirm and summarize the rationale behind consistencies
 Ask raters to articulate the rationale behind inconsistencies
 Review the scoring criteria
 Encourage discussion
RECONCILING INCONSISTENT SCORES
 Descriptive rubrics: Can the criteria or descriptions be revised
in a way that produces agreement?
 Strive for natural delineations in performance that reflect discernable
steps or benchmarks in the development of proficiency
 Rating scales: Can the criteria be revised in a way that reduces
the opportunity for rater bias
 Strive for consensus or at least democratic agreement
 If disagreement persists, change course
 Consider breaking to calculate percentages of agreement
•
•
May help raters calibrate their judgments by seeing how their scores
compare to those of other raters
Calculate the percentage of agreement between pairs of raters and then
calculate the mean of the percentages overall (shown on next slide)
 Consider asking for all to commit to a democratically established
convention for scoring in this particular context
•
Those who are the outliers agree to disagree but nonetheless concede to
score in accordance with the majority’s bias for the sake of consistency
IF THE PROCESS GETS STUCK
Part of Reconciliation
Raters Compared
Total Items
1&2
1&3
Number of
Agreements
3
6
7
7
Percentage of
Agreement
14%
86%
1&4
2&3
2&4
6
4
4
7
7
7
86%
57%
57%
3&4
Overall
7
30
7
42
100%
71%
PERCENTAGE OF AGREEMENT
How to Calculate It
FINAL NOTES
 Rubric norming is a means for creating solidarity among
the faculty regarding what the shared goals are (as
reflected in the student learning outcome statements)
and what student performance looks like when the
outcomes are partially versus wholly achieved.
 Rubric norming is most effective when assessment is
steeped in an ethos of inquiry, scholarly analysis, and
civil academic discourse that encourages faculty
participation in decision-making.