How to Norm Rubrics
Download
Report
Transcript How to Norm Rubrics
HOW TO NORM RUBRICS
Ursula Waln, Director of Student Learning Assessment
Central New Mexico Community College
WHAT IS A RUBRIC?
A rubric is a scoring guide such as:
A checklist
A rating scale
A matrix or list containing descriptions of student work
characteristics at different levels of sophistication, a.k.a. a
descriptive rubric
•
•
Holistic (containing performance levels but not separating out criteria)
Analytic (providing descriptions for each criterion at each
performance level)
Addressed
important
details
Organized
information
logically
Used proper
writing
mechanics
(3)
Proficient
(2)
Developing
Consideration of Diverse Points of View
Developed
key ideas
(1)
Sanitized hands
Verified the
patient’s fasting
status
Asked about
latex sensitivity
Selected
appropriate
gloves and
tourniquet
Assembled
necessary
supplies
Positioned the
patient properly
Descriptive Rubric
Rating Scale
Beginning
Checklist
Beginning
Developing
Proficient
(1)
(2)
(3)
Wholly
dismisses or
disparages
points of view
that diverge
from own
worldview
Identifies
valid
components
of differing
perspectives
but responds
in accordance
with own
worldview
without
reflection
Analyzes the
complexity
and validity
of differing
perspectives
and reevaluates own
perspectives
in light of
alternative
worldviews
With the exception of checklists, rubrics are used to lend a
level of objectivity to evaluation that is inherently subjective
Checklists are for use when the demonstration of learning either
is or is not present, with no in-between degrees of manifestation
•
Checklists do not require norming
Rating scales are the most subjective because they rely on the
scorer’s interpretation of the performance-level headings
Descriptive rubrics can essentially eliminate subjectivity by
clearly identifying indicators of distinct levels of performance
OBJECTIVE VS. SUBJECTIVE SCORING
And Selecting the Right Tool
WHY NORM A RUBRIC?
Rubric norming is called for when more than one
person will be scoring students’ work and the results
will be aggregated for assessment of student learning
To develop shared understanding of the outcome(s) assessed
To discover any need for editing of the rubric
To develop scoring consistency among raters
•
•
Minimize the variable of individual expectations regarding rigor
Minimize potential for differences in interpretation of criteria tied
to identification of performance levels
NORMING RUBRICS OR NORMING RATERS?
Norming rating scales = developing consensus among raters
What the different performance levels are intended to capture
What level of rigor should be applied in distinguishing the levels
All who will be raters should be involved in the norming session(s)
Norming descriptive rubrics = perfecting the rubric
The better written the rubric, the less possibility it allows for
differences in interpretation
•
•
Identify and fix gray areas and ambiguities in the rubric
Reduce need for scorers to conform to a group standard for interpretation
WHAT YOU’LL NEED
A rubric
A facilitator
Some work samples
Student learning outcome statements to which the rubric is tied
An outline of the steps of the norming process
Raters (the faculty who will be doing the scoring)
SELECTING WORK SAMPLES
Real student work or mock-up samples
If using real student work, redact any identifying information
Select samples that demonstrate different performance levels
Plan to have 1 to 3 for each of 2 to 4 scoring sessions
•
Determine number based on the time and complexity of scoring
If the rubric contains multiple criteria, select samples that display
differing levels of performance on differing criteria (i.e., samples
that are neither all good nor all bad)
OUTLINE OF THE STEPS
I. Orientation
II. Discuss levels and rating criteria/thought processes
III. Score the samples
IV. Compare scores
V. Discuss (and possibly modify the rubric)
VI. Repeat above two steps as needed until consensus is reached
ORIENTING THE RATERS TO THE PROCESS COMPONENTS
The student learning outcome(s) being assessed
The purpose of the norming session
The rubric itself
How it came to be
Its intended alignment to the SLO(s)
Its intended use
An outline of the process
Model the thought processes involved in using the rubric
Entertain perspectives regarding:
The number of performance levels and their headings
The construct validity of the criteria that have been identified
Perceived distinctions between performance levels
Perceptions regarding how the faculty think most of the student
work will be scored and the discriminative value of the rubric
DISCUSSING LEVELS & CRITERIA
Part of the Orientation
SCORING THE SAMPLES
All raters score the same 1 - 3 samples concurrently
Raters refrain from discussing the works and/or their scores
during the scoring sessions
Each time, samples provide a range of skill demonstration
Start with the most straight-forward samples and work up to
those that require more refined decision-making
COMPARING SCORES
Look for consistencies and inconsistencies
Confirm and summarize the rationale behind consistencies
Ask raters to articulate the rationale behind inconsistencies
Review the scoring criteria
Encourage discussion
RECONCILING INCONSISTENT SCORES
Descriptive rubrics: Can the criteria or descriptions be revised
in a way that produces agreement?
Strive for natural delineations in performance that reflect discernable
steps or benchmarks in the development of proficiency
Rating scales: Can the criteria be revised in a way that reduces
the opportunity for rater bias
Strive for consensus or at least democratic agreement
If disagreement persists, change course
Consider breaking to calculate percentages of agreement
•
•
May help raters calibrate their judgments by seeing how their scores
compare to those of other raters
Calculate the percentage of agreement between pairs of raters and then
calculate the mean of the percentages overall (shown on next slide)
Consider asking for all to commit to a democratically established
convention for scoring in this particular context
•
Those who are the outliers agree to disagree but nonetheless concede to
score in accordance with the majority’s bias for the sake of consistency
IF THE PROCESS GETS STUCK
Part of Reconciliation
Raters Compared
Total Items
1&2
1&3
Number of
Agreements
3
6
7
7
Percentage of
Agreement
14%
86%
1&4
2&3
2&4
6
4
4
7
7
7
86%
57%
57%
3&4
Overall
7
30
7
42
100%
71%
PERCENTAGE OF AGREEMENT
How to Calculate It
FINAL NOTES
Rubric norming is a means for creating solidarity among
the faculty regarding what the shared goals are (as
reflected in the student learning outcome statements)
and what student performance looks like when the
outcomes are partially versus wholly achieved.
Rubric norming is most effective when assessment is
steeped in an ethos of inquiry, scholarly analysis, and
civil academic discourse that encourages faculty
participation in decision-making.