Transcript Evaluation and Assessments PHED 6321 – Pedagogy: The Art

Approaches to
Effective and Efficient
Assessment and
Grading
Carol A. Kominski, Ph.D.
Assessment Specialist
Authentic Assessment
• If you must take a chauffeur-driven trip, what do you
prefer when you have only two choices?
– A driver who passed the written portion of the licensing
exam but failed the driving portion
– A driver who passed the driving portion of the licensing
exam but failed the written portion
Efficiency*Continuum
Portfolios, clinical
applications,
reflection, research
reports
Case-based
analyses, oral
presentations,
demonstrations
Short
written or Fill in the
blank
oral
responses questions
Low
*Efficiency = ease and speed of grading.
Multiple
choice,
matching, T-F
High
Authenticity*Continuum
Multiple choice,
matching, T-F
Low
Short
written or
Fill in the blank oral
questions
responses
Case-based
analyses, oral
presentations,
demonstrations
Portfolios, clinical
applications,
reflection, research
reports
High
*Authenticity = degree to which requirement is similar to that likely to be
encountered in workplace or professional practice.
Assessment Challenges
• How can efficient assessments be more authentic?
(Strategies to Assess Large Classes)
• How can authentic assessments be more efficient?
(Approaches to Effective and Efficient Assessment
and Grading)
Potential Authentic Assessments
Written Work
Oral Communication
Demonstration
• Short answer or essay test/assignment
• Research report
• Case Study Analysis
• Class presentation or speech
• Debate
• Conversation
• Specific research procedure or technique
• Clinical procedure, e.g. blood sampling
• Client or patient interaction
Disposition or Behavior
• Surveys
• Records of participation
• Unobtrusive measures of behavior
Comprehensive/Hybrid
• E-Portfolio
• Instructional unit
• Reflection paper
Web Resource on Authentic Assessment
Authentic Assessment Toolbox – provides
background, explanations, examples.
• http://jfmueller.faculty.noctrl.edu/toolbox/wh
atisit.htm
How to make authentic assessment
manageable and efficient
• Rubrics (otherwise known as Standards, Criteria, Measures)
– Multiple dimensions
– Scale Placement Categories
• Usually 3 to 5 choices on continuum
• Only numbers, e.g, 1 to 5 scale
– Poor, Fair, Good, Excellent
– Novice, Developing, Accomplished
– Doesn’t Meet, Meets, Exceeds Expectations
– Your own creation
• Checklists (Two Categories)
– Yes or No/True or False/Off or On
– Can be complemented with
• Student Involvement
• Peer assessment
• Teaching assistants
What are rubrics?
“A scoring tool that lays out the
specific expectations for an
assignment. Rubrics divide an
assignment into its component parts
and provide a detailed description of
what constitutes acceptable or
unacceptable levels of performance
for each of those parts.” Parts are
– Task Description.
– Scale.
– Dimensions.
– Dimension Descriptions.
Source: Stevens, Danielle D. and Levi,
Antonia J. (2005). Introduction to
Rubrics, 3.
“A scoring tool for qualitative rating of
authentic or complex student work. It
includes criteria for rating important
dimensions of performance, as well as
standards of attainment for those
criteria. The rubric tells both instructor
and student what is considered
important and what to look for when
assessing.”
Source: Jonsson, A. & Svingby, G. (2007).
The use of scoring rubrics: Reliability,
validity, and educational consequences.
Educational Research Review. 2, 130-144.
Why rubrics are good assessment practice
•
•
•
•
•
•
Timeliness
Detailed Feedback
Critical Thinking
Communication
Refinement of teaching skills
Level playing field
Source: Stevens, Danielle D. and Levi, Antonia J. (2005). Why use rubrics?
Introduction to Rubrics, 17-28.
Steps in Constructing a Rubric
• Determine learning objectives.
• Outline expectations.
– Use results of previous assignments, if possible.
• What’s good?
• What’s acceptable?
• What’s unacceptable or poor?
– Start with what others have done.
• Develop rubric dimensions, e.g.,
–
–
–
–
Organization
Persuasiveness of arguments
Research support
Quality of writing
• Fill in the rubric.
Rubric Features
• Rubrics can be created in a variety of forms and levels of
complexity, however, they all contain three common features
which:
– focus on measuring a stated objective (performance, behavior, or
quality).
– use a range to rate performance.
– contain specific performance characteristics arranged in levels
indicating the degree to which a standard has been met.
Source: Pickett, N. and Dodge, B. (2007). Rubrics for Web Lessons.
http://webquest.sdsu.edu/rubrics/weblessons.htm
Rubrics
Do’s and Don’ts
Do
Don’t
• Describe performance levels
clearly.
• Solicit advice from others (other
faculty, teaching assistants, even
students).
• Use anchor examples from
previous assessments, if possible.
• Measure consistency (over time
and raters).
• Revise as needed for consistency,
clarity, comprehensiveness, ease
of use.
• Make sure relative emphasis (or
points) is placed on high priority
criteria.
• Assume that students
understand the rubric
without discussion and
explanation.
• Make rubric a counting tool
which eliminates
professional judgment.
• Allow rubric to set too low a
limit on student
achievement.
• Use surprise rubrics shared
with students only after
assignment submission.
Checklists
• A simple type of rubric
• For either-or classification.
• Good for determining compliance with format
and content requirements.
• Typically used for lower order items.
• Examples
 Abstract
 APA format
 Two examples
 On time submission
Example of Combination Rubric
Student Involvement Models
Presentation
Feedback
Pass-the-Hat
Post-it
4 X 4 Model
• just sharing
• rubric can be changed
• students help fill in dimensions
• students create dimensions and fill them in
• students participate in all stages of rubric creation
Source: Stevens, Danielle D. and Levi, Antonia J. (2005). Rubric Construction and the
Classroom. Introduction to Rubrics, 49-64.
Peer Assessment
• Designed for mature students.
• Engages learners in authentic learning.
• Students develop assessment criteria.
• Students assess others’ learning, grade work, and receive
feedback.
• Tutor checks to ensure quality.
Source: Juwah, C. (2003). Using Peer Assessment to Develop Skills and Capabilities.
Journal of the United States Distance Learning Association, January 2003, 39-50.
Teaching Assistants
• Collaboration helps in
• Understanding what instructor is trying to
accomplish.
• Modeling good teaching practice.
• Obtaining insight into common student
misunderstandings and issues.
Web Resources: Rubric Examples from Higher
Education
• American Association for Learning in Higher Education
(AALHE)
– http://course1.winona.edu/shatfield/air/rubrics.htm
• California State University at Bakersfield
– http://www.csub.edu/TLC/options/resources/handouts/Rubric_
Packet_Jan06.pdf
• Loyola Marymount University
– http://www.lmu.edu/about/services/academicplanning/assessment/A
ssessment_Resources/Rubrics/Example_Rubrics.htm
• California State University at Fullerton Business School
– http://business.fullerton.edu/centers/CollegeAssessmentCenter
/RubricDirectory/other_rubrics.htm
Rubrics: Examples from Pharmacy, Physician’s
Assistant, Public Health programs
• Pharmacy SOAP note rubric
– http://www.rcampus.com/rubricshowc.cfm?code
=Z8WACB&sp=true&
• Physician’s assistant physical history rubric
• http://www.rcampus.com/rubricshowc.cfm?code=W4A5C
2&sp=true
• Public Health Rubric
• (http://www.tamhsc.edu/about/accreditation/sacs/CAREQEP.pdf) See next page.
Other Rubrics: A Few Examples
• Ethics
– http://www.personal.psu.edu/faculty/d/x/dxm12/
n458/sample_case_rubric.htm
• Case analysis
– http://academic.scranton.edu/department/assess
ment/ksom/case-analysis-rubric.pdf
• Critical thinking
– http://course1.winona.edu/shatfield/air/Critical%
20Thinking-long.pdf
What’s wrong with this rubric?
Challenge: Design rubrics for assignments like the
following.
Assignment 1. Describe consequences of high sodium consumption
on the human body and its possible relationship to health problems
and morbidity.
Assignment 2. Demonstrate procedures for determining whether a
patient has a potential heart problem.
Assignment 3. Describe, evaluate, and compare at least two
potential solutions to reduce the proportion of the population who
do not have health insurance.
More Support Needed
• Contact Center for Learning and Development
(CLD) for
– Assessment Design (Carol Kominski Ext. 2942)
– Instructional Design (Kun Huang Ext. 2941)
– Blackboard (AZ Bashet Ext. 2943)
– Psychometric analysis (Kevin Kalinowski Ext. 2543)
– Other support (Director Vanneise Collins Ext.
5056)