Transcript Rubrics for oral assessment

Designing Rubrics For Classroom
Assessment
Professor Timothy Farnsworth, CUNY Hunter College
Communicative Assessment Is Hard!



We are making inferences about what people can and can’t
get done in the real world, not:
What grammar or vocabulary they can choose from 4
options on a MC test
Tasks must be realistic but can’t be TOO realistic:




2
Time constraints
Stakes are different
Interlocutors are different
Setting, format usually different
Speaking Assessment Challenges
Speaking is the hardest skill to assess
Impermanent -> must be rated on the spot or recorded
Interactive -> Real communication takes two persons
Nonverbal -> Much real communication takes place
without language use
Topical -> Students’ knowledge base affects their speaking
output





3
Oral Assessment / Testing Tasks
Must get students involved in speaking!
In communication, many things happening






Listening
Nonverbal channeling
Decisions made about what to say, what not to say
Students practice avoidance strategies, game the system, don’t
take risks
Assessment tasks take many forms

4
Types of Assessment Tasks
Many options – some common ones:
Interview, either scripted or unscripted
Picture description
Role Play
Pair or group tasks
Almost any classroom task, really!






5
Sample task:
Pair oral
assessment

Each student has a
different picture;
communicate to find
similarities and
differences
6
Scoring tasks and communicating results
We need some way to turn the “observation” into a “score”
or “grade”
Usually done via numbers, can be done as letters, smiley
faces, etc. – doesn’t matter
Should be clear, simple
Should be “reliable” -> repeatable, dependable
Should be “valid” -> meaningful
Shouldn’t cause any harm






7
Types of scoring rubric
Holistic
Analytic
Task Fulfillment (“primary trait”)
Performance decision trees
Checklists





8
Score
Description
Communication almost always effective: task performed very competently
Holistic
Rubrics
Functions performed clearly and effectively
60
response to audience/situation
OneAppropriate
overall
score
tied to a “performance descriptor”
Coherent, with effective use of cohesive devices
of linguistic
features
MostUsefamiliar
type
of almost
rubricalways effective; communication not affected by
minor errors
Assign score of the description that best matches
…..
performance
Communication somewhat effective: task performed somewhat competently
Problem:
Performance doesn’t match description at the
Functions performed somewhat clearly and effectively
same
level inappropriate
all areas
Somewhat
response to audience/situation
Somewhat coherent, with some use of cohesive devices
Advantage:
Simple
to somewhat
use andeffective;
understand
Use of linguistic
features
communication sometimes affected



50
40


by errors
30
…
20
No effective communication: no evidence of ability to perform task
No evidence that functions were performed
…
9
Analytic Rubrics
Level
0 

1 



Accuracy
Fluency
Pronunciation
Two
moreresponse;
categories
of scores
Noor
structured
Speech
is halting,
Wholly or partly unintelligible
single words only
long pauses
due to pron. Errors
Usually
based around linguistic
categories
such as
pronunciation, grammar, etc.
Also
quite
common
Multiple
inaccuracies
in
Pauses occur but
Pron. Errors increase difficulty
vocabulary choice,
are not overly long, for hearers
Assign
scores
for eachspeaker
category
per performance
sentence
structure
does not
inordinately ‘word
Problem: Can be cognitively
challenging for raters
search’
Advantage: provides more detailed information
2
Few major errors wrt
vocab or SS
10
Normal or nearnormal speed and
flow of speech
Pronunciation is native-like or
does not increase difficulty for
listeners
Task Fulfillment Rubrics
Score
 UsuallyDescription
look like holistic rubrics
4 InsteadCandidate
was polite,
friendly,
guest
successfully,
of describing
language,
theyhelped
describe
whether
the
to the
complaints
studentresponded
completed
“task” appropriately, asked for
clarification, etc.

3
2
1


0
Usually used in role play type assessments
Useful …
in testing for employment, other settings
Useful …
when there is a specific set of tasks that involve
language
and canappeared
be specified
Candidate
rude, could not help guest
Example
rubric fordid
hotel
employee
successfully,
not respond
to testing:
complaints
appropriately,
could
ask to
for respond
clarification,
etc.
Job applicant
is tested
fornot
ability
to guest
requests and complaints, answer telephone, etc. in role
play situation,
No response
11
Performance Decision Trees





Fairly new to the scene (Fulcher 2011)
Use a flowchart design
Good for clear, simple scoring when the task is very
consistent / clear
Give points for each successful “yes” on the flowchart
Lets the test raters answer very specific questions
12
13
Self / Peer Assessment





Students have some good ideas about their and their
classmate’s performances
In a busy classroom, teacher cannot hear everyone
Students can use any feedback they get
Training students to listen carefully and help each other is
good practice in general
But: Students are not always good evaluators of their peers



May not catch pronunciation or other important mistakes
May be LESS proficient than the person they are supposed to
assess
Only ask them to assess in areas they can definitely “do” –
yes / no checklists are great for this
14
Checklist Rubrics

Phrased very simply like a can-do statement:



“Student can understand and use vocabulary for classroom objects.Yes
/ No / Partially
“Student can use formal language in giving oral presentations” Yes / No
/ Partially
Can be broken into parts:



“Student understands WH questions” Yes / No / Partially
“Student responds to WH questions using appropriate grammar and
vocabulary” Yes / No / Partially
“Student responds to WH questions at length and returns questions
appropriately” Yes / No / Partially
15
Checklists for student self-evaluation





Self Evaluation – teacher collects evidence
Checklists are simple and clear (example)
List main objectives for the week, create checklist for
students
Have them do and turn in anonymously
Use the info to plan review sesssions in future
16
Keys to consistent, fair scoring
1: KISS principle for tasks







Keep It Simple, Sweetheart!
Don’t use any more complicated assessment than you
really need
One or two tasks is enough as long as you are getting the
right kind of interaction
Make sure you get “enough speech” to make a rating using
the right rubric
Make a system that everyone can easily understand
Good speaking testing usually can be done in less than five
or ten minutes
More “high stakes” testing takes longer, up to an hour
17
Keys to consistent, fair scoring
2: Use the simplest rubric that works


I like checklists or performance trees for classroom and
self assessments
I like analytic rubrics for more high-stakes testing



They give information on what things the student needs to
work on
Holistic rubrics don’t provide much information for students
I don’t like task fulfillment rubrics for most situations


18
It’s not clear from the score what the student needs to work
on
It’s sometimes possible to do a task using little language; the
task fulfillment setup still allows high scores
Keys to consistent, fair scoring
3: Write the rubric so students can understand it



Use plain and simple language
Use “I can / can’t do X” or “You did X” when possible
Explain the scores and the rubric to them before the test
–




no reason for it to be a secret
They should understand why they got their score and what
they can work on next time
Do not use a self assessment for a grade
Do not use peer assessment for a grade
19
Your assignment: Write a scoring rubric



Find a partner next to you
Together, pick one rubric style:
Holistic, analytic, decision tree, checklist, task fulfillment
Think about students in a typical class you teach:




20
What level are they?
What do you need to learn more about in terms of their
speaking skills?
Write a rubric with a few levels and clear descriptions
Compare your rubric with a pair near you – does it look
similar? Why or why not?
Sample task:
Pair oral
assessment

Each student has a
different picture;
communicate to find
similarities and
differences
21