Transcript Assessment in Education - University of Arkansas for

Assessment in Education
Patricia O’Sullivan
Office of Educational Development
UAMS
Last Month Dr. Allen described a
situation about an individual asking
her how to get to Freeway Medical
Center. How would you determine
how successful Dr. Allen was?
Objectives




Provide terminology and principles
fundamental to assessment
Introduce various assessment
approaches
Review tips for a common
assessment: Multiple Choice
Questions
Review elements for a less common
assessment: Portfolio
What is Assessment?


Appraising performance through
measurement to make an
evaluation
For what purpose?


Formative
Summative
What Assessments are Used at
UAMS?
What Makes a Good
Summative Assessment?




Fair and practical
Based on appropriate criteria that
are shared with the learners
Valid
Reliable
Validity

Key to the interpretability and
relevance of scores




Do the scores have a plausible
meaning?
What are the implications of
interpreting the scores?
What is the utility of the scores?
What are the actual social
consequences?
Validity Evidence for Proposed
Interpretation

Content/Construct Evidence



Content matches the objectives &
teaching
Score is shown to be meaningful
Criterion/Predictor Evidence

Score relates to future outcomes
Reliability

Assessment data must be
reproducible

Written tests
•
•
Internal consistency
Cronbach’s alpha or KR-20 (values
range 0-1)
Test-retest
Correlation ranging from 0-1
Reliability (cont.)

Rater-based assessments


Interrater consistency or agreement
(generally correlation coefficient or
intraclass correlation)
OSCE & performance assessments

Generalizability Theory
Improve Validity





Match assessment with objectives
Increase the sample of objectives
and content on the assessment
Review blueprint (or produce one)
Use test methods appropriate to
objectives
Ensure test security
Improve Reliability





Clear questions
Appropriate time allotted
Simple, clear and unambiguous
instructions
High quality scoring
Increase number of observations or
questions
Norm or Criterion Referenced

Norm-referenced (relative standard)


Use the results of the assessment to set the
standards (e.g. proportion to receive each
grade); performance judged by comparison
with others
Criterion-referenced (absolute
standard)

Learners achieve some minimal standard of
competency
Standard Setting Procedures

Judgments based on



Holistic impressions of the exam or
item pool
Content of individual test items
Examinee’s test performance
Anghoff Standard Setting
Procedure




Judges are instructed to think of a group
of “minimally acceptable” persons
For each item estimate the proportion of
this group that will answer item correctly
Sum up proportions to get the minimum
passing score for a single judge
Average across judges
Assessment Methods


ACGME Toolbox
Additional methods

Other written assessments
•
•
•
•


Essay
Short answer and computation
Patient management problems
Modified essay questions
Create a game
Self-assessment
Matching Assessment Method
and Objectives

Examine ACGME matrix
Alternative Testing Modes




Take-home tests
Open-book tests
Group exams
Paired testing
Reducing Test Anxiety





Make the first exam relatively
easier
Give more than one exam
Give advice on how to study
Encourage study groups
Give a diagnostic test early
Reducing Test Anxiety



Before a test explain format
Provide sample items
Provide a pool of final items with
syllabus
Technology and Assessment

Computer based testing


Use of images and video clips
Computer Adaptive Testing (CAT)
Take a Test
Multiple Choice Questions:
Writing Tips

http://www.nbme.org/about/itemwriting.asp

Simple suggestions:




Design for one best answer
Place hand over stem. Can you
guess the answer?
Use a template
Attend to Graveyard of Test Items
Template Examples


A (patient description) is unable to
(functional disability). Which of the
following is most likely to have
been injured?
Following (procedure), a (patient
description) develops (symptoms
and signs). Laboratory findings
show (findings). Which of the
following is the most likely cause?
Graveyard



B-type: matching heading with
words or phrases (can use heading
more than once)
D-type: complex matching—two
step process
K-type: stem with four options and
choices are


A=1,2,3 only B=1,3 only C=2,4 only
D=4 only, E=all are correct
Portfolios
Portfolios

Definition

A purposeful collection of student
work that exhibits to the student
(and/or other) the student’s efforts,
progress, or achievement in (a) given
area(s).

This collection must include student
participation in




Selection of portfolio content
The criteria for selection
The criteria for judging merit
Evidence of student
reflection
Portfolios

Learning



Showing progress
Showing effort
Showing
remediation

Assessment

Showing
competence
Presentations
Logs
Case Records
Evaluations
Criteria
______
______
______
Criteria
Portfolio
Using Portfolios the Learner
Engages in Assessment



Making choices
Critically self-examining
Explaining what they are doing
Well-structured Portfolio





Menu
Selection criteria
Scoring rubric
Benchmarks
Trained raters
Determine Reliability and Validity

Reliability: Generalizability theory


Found needed 6 entries with 2 raters
Validity: number of studies





Resident performance across years
Resident performance in areas
Resident perception
Resident performance when program made
an intervention
Correlation of portfolio performance and
other areas
10-Step Approach
1.
2.
3.
4.
5.
Define the purpose
Determine competencies to be
assessed
Select portfolio materials
Develop an evaluation system
Select and train examiners
10-Step Approach
6.
7.
8.
9.
10.
Plan the examination process
Orient learners
Develop guidelines for decisions
Establish reliability and validity
evidence
Design evaluation procedures