Transcript Developing Rubrics - Central New Mexico Community College

Creating
Descriptive Rubrics
for Educational Assessment
Ursula Waln, Director of Student Learning Assessment
Central New Mexico Community College
Why Create Rubrics?

To communicate learning expectations to students.

To facilitate fairness and consistency in evaluation
of student learning.

To establish a common framework for assessing
student learning when using diverse measures.
Choose a Layout
To Fit Your Purpose
Two Main Types of Descriptive Rubrics
Holistic

Provides comprehensive
descriptions of each level
of performance.

Useful for quick and
general assessment and
feedback.

Descriptions may be
organized in columns or
rows.
Analytic

Breaks out criteria for
distinguishing between
levels of performance on
each criterion.

Useful for detailed
assessment and feedback.

Descriptions are organized
in a matrix.
HOLISTIC RUBRIC FOR RUBRICS (in columns)
Highly Effective
3
Useful
2
Rudimentary
1
Focuses on knowledge and skills that
confer life-long value through application
to novel or complex situations, clearly
addressing key criteria that 1) comprise
the desired learning outcome, and 2) can
be demonstrated through a variety of
tasks. Differentiates benchmark levels
inherent to development of the
competency, thereby facilitating valid
assessment of both formative and
summative progress. Clearly and concisely
communicates the nuances between
differing levels of proficiency, and does so
in terms of positive evidence.
Clearly addresses key
criteria that comprise the
desired learning outcome.
Organizes performance
characteristics within
progressive categories that
accurately reflect
observable stages of
development. And, clearly
describes observable
characteristics that
demonstrate each level of
performance on each
criterion.
States learning
outcomes in terms of
broad instructional
goals and/or specific
learning tasks; presents
performance
characteristics in terms
of correctness,
quantity, or other
grading criteria;
constitutes a Likert
scale.
HOLISTIC RUBRIC FOR RUBRICS (in rows)
Highly Effective (3) Focuses on knowledge and skills that confer life-long value through
application to novel or complex situations, clearly addressing key
criteria that 1) comprise the desired learning outcome, and 2) can be
demonstrated through a variety of tasks. Differentiates benchmark
levels inherent to development of the competency, thereby facilitating
valid assessment of both formative and summative progress. Clearly
and concisely communicates the nuances between differing levels of
proficiency, and does so in terms of positive evidence.
Useful (2)
Clearly addresses key criteria that comprise the desired learning
outcome. Organizes performance characteristics within progressive
categories that accurately reflect observable stages of development.
And, clearly describes observable characteristics that demonstrate
each level of performance on each criterion.
Rudimentary (1)
States learning outcomes in terms of broad instructional goals and/or
specific learning tasks; presents performance characteristics in terms
of correctness, quantity, or other grading criteria; constitutes a Likert
scale.
ANALYTIC RUBRIC FOR RUBRICS
Criteria
Highly Effective
3
Useful
2
Rudimentary
1
CompetencyBased
Focuses on knowledge and skills that
Clearly addresses key
confer life-long value through application criteria that comprise the
to novel or complex situations, clearly
desired learning outcome.
addressing key criteria that 1) comprise
the desired learning outcome, and 2) can
be demonstrated through a variety of
tasks.
States learning
outcomes in terms of
broad instructional
goals and/or specific
learning tasks.
Representative
Differentiates benchmark levels inherent
to development of the competency,
thereby facilitating valid assessment of
both formative and summative progress.
Organizes performance
characteristics within
progressive categories that
accurately reflect
observable stages of
development.
Presents
performance
characteristics in
terms of correctness,
quantity, or other
grading criteria.
Descriptive
Clearly and concisely communicates the
nuances between differing levels of
proficiency, and does so in terms of
positive evidence.
Clearly describes
observable characteristics
that demonstrate each
level of performance on
each criterion.
Constitutes a Likert
scale.
Identify the Performance Criteria
What Will Be Assessed
First, Focus on Competencies

Whether designing a rubric for a single assignment
or for assessing course or program outcomes,
focus on competencies, not specific tasks.
 Competencies
involve knowledge and skills that confer
life-long value through application to novel or complex
situations.
 Tasks
are used to collect evidence of achievement of
the competency, but a competency-based rubric can be
applied to a variety of tasks.
Examples of Competencies vs. Tasks
Competencies
Tasks

Apply knowledge of social
systems to analyze
contemporary global issues.

Discuss the role of capitalism
in the Chinese construction
boom.

Frame problems within
differing ethical
perspectives.

Critique an opinion article
for political bias.

Correctly identify websites
that are suspect choices as
sources for academic
research.

Write an e-mail to inquire
about a job opening.

Evaluate information and
sources critically.

Employ contextually
appropriate etiquette within
various forms of electronic
communication.
Design with Context In Mind

Focus on competencies that align to pertinent:
 Standards
 Student
(educational, industry, program)
learning outcome statements (program, course)
 Instructional
 Learning
goals
objectives
Next, Derive Criteria from Competencies
Competencies
Criteria

Broad descriptions of
desired learning
outcomes.

Standards on which
judgments about level of
competency can be made.

What you want the
student to be able to
do upon completion of
the lesson, course, or
program.

Characterizing traits
(characteristics) of the
competencies.

What is needed to
demonstrate the
competencies.
From Competencies to Criteria

Some competencies are straight-forward enough
to be used as criteria in rubrics. For example:
 Competency:
 The

student will be able to…
Apply the commutative property in computation.
 Criterion:
 To
demonstrate application of the commutative property, the
student needs to…

Apply the commutative property in computation.

Some competencies are overarching and/or
complex and need to be broken down for
analytic evaluation. For example:
 Competency:
 The

teacher will be able to…
Create effective rubrics.
 Criteria:
 To
be effective, the teacher’s rubrics need to be…

Competency-based

Representative

Descriptive
Break Down Compound Competencies

This competency represents a combination of
abilities:
 Produce
audience appropriate communication that
displays consideration of ethical principles and diverse
points of view.

Compound competencies are more easily
evaluated when broken down into component
criteria. Using the above example:
 Audience
appropriateness
 Consideration of ethical principles
 Consideration of diverse points of view
Consider Differences in Manifestation

Some competencies call for synthesis of several
related manifestations of learning. For example:
 The
competency “Employ contextually appropriate
etiquette within various forms of electronic
communication” involves:
 SKILLS/ABILITIES
related to language and register-based
code-switching.
 KNOWLEDGE/AWARENESS
of etiquette associated with
different contexts.
 USES/APPLICATIONS
of electronic communication media.
 BEHAVIORS/DISPOSITIONS
related to courtesy and respect.

Evaluation can be facilitated by breaking
different manifestations of learning out into
separate criteria. For example:
 Competency
 The

(from previous slide):
student will be able to…
Employ contextually appropriate etiquette within various
forms of electronic communication.
 Criteria:
 To
show proficiency, the student needs to demonstrate…

Fluency in code-switching

Familiarity with contextual expectations

Application of electronic media

Courtesy
Can One Part Be Done Without the Other?

“Demonstrate leadership and integrity.”
 Yes!
We can find examples of strong leaders with little
integrity and people of high integrity with very poor
leadership skills. Therefore, for analytic evaluation
purposes, these should probably be separate criteria:

 Demonstrate
leadership.
 Demonstrate
integrity.
“Analyze and evaluate college-level texts.”
 This
one is not so clear. Evaluation requires analysis,
and effective analysis requires a modicum of evaluative
skill. It might be best to leave these together unless
you intend to evaluate them as separate skills.
Is the Proficiency Multi-Faceted?

“Evaluate information and sources critically.”
 Yes,
this consists of two separate abilities:
 Evaluate
information critically.
 Evaluate
sources critically.
 And,
each of these could be further broken down:
 Evaluate
information critically → recognize manipulative
rhetorical devices, recognize bias, assess contradictory
information, identify failures in logic, analyze alternative
points of view, etc.
 Evaluate
sources critically → analyze context, infer motive,
assess credibility, etc.
How Far Should You Break Things Down?

Consider the purpose of the rubric.
 Program-level
 Course-level

assessment uses broadly defined criteria.
assessment uses more specific criteria.
Consider performance evaluation.
 Break
apart criteria if you would use separate measures
to evaluate the components.

Consider complexity.
A
component that has nuanced and/or distinguishable
performance levels probably needs to be separated out.
How Far Is Too Far?

Rubrics usually have more than two performance
levels, so use ‘all or nothing’ as a litmus test for
too much specificity. For example:
 “Differentiate
between collective and individualistic
societies.”
 If
a student either is or is not able to identify a basic difference
(all or nothing), consider using a broader criterion, such as
“Describe societal differences that contribute to diverse world
views.”
 On
the other hand, if a student might describe a variety of
characteristic differences and do so with nuanced degrees of
accuracy, then the criterion should probably stand alone.
Phrasing of Criteria

Criteria are usually written as phrases and are
formatted according to personal taste.
 Some
examples of different formats:
 Employs
 Employ
 Fluent
code-switching fluently
code-switching fluently
code-switching
 Code-switching
 Fluent

fluency
in code-switching
For rubric clarity and cohesion, pick a format and
be consistent.
Create the Headings for a Matrix
Analytic Rubrics
Criteria & Performance Levels

In analytic rubrics, the convention is to use the
left-hand column for listing the criteria.
 You
can leave the top cell blank or enter a heading,
such as “Criteria.”

The tops of the remaining columns hold the
headings for the various performance levels.
Basic Layout
Level
Heading
1
Criterion 1
Criterion 2
Criterion 3
Criterion 4
Level
Heading
2
Level
Heading
3
Determine How Many Columns You Need

How many performance levels do you want to
describe?

The usual is 3 to 5, but some advise using an even number
(to avoid a “central” tendency in scoring).

Do you want to include an exceptional level?

Do you want to include a scoring column?


Helpful if the rubric will be used as a scoring instrument.
Do you want to include unlabeled columns with
numerical values to allow for in-between scoring?
Level
Heading
1
2
Level
Heading
3
Level
Heading
4
5
Exceptional
Level
Heading
6
Criterion 1
Description
Cr. 1 Lvl. 1
Description
Cr. 1 Lvl. 3
Description
Cr. 1 Lvl. 5
Description
Cr. 1 Lvl. 6
Criterion 2
Description
Cr. 2 Lvl. 1
Description
Cr. 2 Lvl. 3
Description
Cr. 2 Lvl. 5
Description Cr.
2 Lvl. 6
Criterion 3
Description
Cr. 3 Lvl. 1
Description
Cr. 3 Lvl. 3
Description
Cr. 3 Lvl. 5
Description Cr.
3 Lvl. 6
Criterion 4
Description
Cr. 4 Lvl. 1
Description
Cr. 4 Lvl. 3
Description
Cr. 4 Lvl. 5
Description Cr.
4 Lvl. 6
Score
Sample Layout with Unlabeled and Scoring Columns
Choose Ascending or Descending Order

The choice of whether to arrange the levels in
ascending or descending order is a matter of
preference.
 Does
one order make more sense to you than the other?
 Does
your topic suggest a certain organizational logic?
 Are
your students accustomed to a certain layout?
 Does
your school or program have a preferred model?
Exceptional
Level
Heading
6
Level
Heading
5
4
Level
Heading
3
2
Level
Heading
1
Criterion 1
Description
Cr. 1 Lvl. 6
Description
Cr. 1 Lvl. 5
Description
Cr. 1 Lvl. 3
Description
Cr. 1 Lvl. 1
Criterion 2
Description
Cr. 2 Lvl. 6
Description
Cr. 2 Lvl. 5
Description
Cr. 2 Lvl. 3
Description
Cr. 2 Lvl. 1
Criterion 3
Description
Cr. 3 Lvl. 6
Description
Cr. 3 Lvl. 5
Description
Cr. 3 Lvl. 3
Description
Cr. 3 Lvl. 1
Criterion 4
Description
Cr. 4 Lvl. 6
Description
Cr. 4 Lvl. 5
Description
Cr. 4 Lvl. 3
Description
Cr. 4 Lvl. 1
Score
Sample Layout with Levels in Descending Order
Compose Performance-Level Headings

Performance-level headings are usually one or
two words each, carefully selected to represent
the developing levels of proficiency.

If the rubric is to be used as a scoring instrument,
include a numeric score value with each heading.

You can decide upon headings early on or after
writing the descriptions.
 Sometimes
 If
it helps to write the descriptions first.
you want to wait, just enter numbers as place
holders.
Sample Headings
Beginner, Amateur, Pre-Professional, Professional
 Beginning, Emerging, Developing, Proficient, Exemplary
 Below Expectations, Satisfactory, Exemplary
 Benchmark, Milestone, Capstone
 Level 1, Level 2, Level 3, Level 4
 Needs Improvement, Acceptable, Exceeds Expectations
 Needs Work, Acceptable, Excellent
 Neophyte, Learner, Artist
 Novice, Apprentice, Expert
 Novice, Intermediate, Advanced
 Rudimentary, Skilled, Accomplished
 Undeveloped, Developing, Developed, Advanced

Emerging
Developing
Proficient
1
2
3
4
5
Criterion 1
Criterion 2
Criterion 3
Criterion 4
Exemplary
6
Score
Example of Headings with Associated Values
Write Descriptions
In the Intersecting Cells
Describe the Target Outcome

Start by describing the highest level of
performance expected.
 Describe

any level that exceeds expectations last.
Descriptions should focus on what is observable.
 What
does it look like (how can you tell) when the
student has achieved the desired learning outcome?
 Try to capture the essence of the outcome, the
important manifestations.
 Use clear and concise language.
 It is okay to use sentence fragments, so long as the
meaning is clear.
What Is Observable?

Internal processes, such as understanding,
appreciating, valuing, respecting, being curious,
and being interested are not readily observable.

So, we have to identify behaviors/actions that
evince those processes, such as explaining,
analyzing, classifying, applying, appraising,
defending, integrating, transforming, etc.

Avoid putting the word demonstrates in front of an
internal process to make it observable (e.g.,
demonstrates understanding or demonstrates
appreciation) unless you truly need a vaguely worded
rubric for broad application and broad interpretation.
Describe the Other Levels

Seek to describe performance breakthroughs or
thresholds that logically distinguish the levels of
achievement.

If you include an exceptional level (“Exceeds
Expectations,” “Outstanding,” etc.), make it
clear the proficiency goes well above and beyond
the learning expectations.
 Scoring
in this column should be rare; otherwise, you
probably need to adjust your learning expectations.
Use Cumulative or Distinct Descriptions
Cumulative

Build upon positive
indicators of
achievement.



What the student does do
versus does not do.
Sophistication of indicators
increases with proficiency.
Lower-level abilities are
assumed to be present
within the more
advanced levels.
Distinct

Each level is described in
full.

Lower levels typically
include negative as well as
positive indicators.

Tend to repeat and add to
lower-level positive
indicators.

Can be tricky to write well
because saying a student
does not do something is
fairly absolute and tends to
restrict interpretation.
Sample Criterion: Discuss Design Concepts
Cumulative
Descriptors



Level 1: Identifies
fundamental design
concepts.
Level 2: Applies design
concepts to describe
works.
Level 3: Applies design
concepts to effectively
critique works.
Distinct Descriptors

Level 1: Identifies but does
not apply design concepts.

Level 2: Identifies design
concepts and applies them
to describe works but does
not effectively apply them
in critiques of works.

Level 3: Identifies design
concepts and applies them
to describe and effectively
critique works.
Allow the Creative Process to Unfold

Revise
 Some
people can clearly envision the whole and
implement it without modification, but most of us
adjust a little here and there as we see the totality
begin to emerge more clearly.

Entertain Ideas
 The
process of creating rubrics often helps teachers
become much more clear in their own minds about
what they want their students to learn, which can
stimulate new ideas about how to promote that
learning.
Rubric for Rubric Component Analysis
Good
Even Better
Visual organization forms a
clear basis for either holistic
or analytic evaluation.
For Greatest Effect
Layout
Layout presents or implies a
matrix with more than one
performance level.
Choice of layout is
appropriate to the intended
use and audience.
Performance
Criteria
Criteria clarify key aspects of Criteria are distinct,
proficiency.
scalable, and measurable.
Criteria are clearly worded
and consistently formatted.
Headings
Headings represent
performance levels, arranged
in either ascending or
descending order.
Headings accurately reflect
developmental processes
associated with the desired
learning.
Word choices are appropriate
to the context in which the
rubric will be used.
Descriptions
Descriptions characterize
proficiency at successive
stages of development.
Higher-level descriptions
relate breakthroughs in
proficiency development.
Descriptions emphasize
positive indicators of
proficiency.