Assessment in Education - University of Arkansas for
Download
Report
Transcript Assessment in Education - University of Arkansas for
Assessment in Education
Patricia O’Sullivan
Office of Educational Development
UAMS
Last Month Dr. Allen described a
situation about an individual asking
her how to get to Freeway Medical
Center. How would you determine
how successful Dr. Allen was?
Objectives
Provide terminology and principles
fundamental to assessment
Introduce various assessment
approaches
Review tips for a common
assessment: Multiple Choice
Questions
Review elements for a less common
assessment: Portfolio
What is Assessment?
Appraising performance through
measurement to make an
evaluation
For what purpose?
Formative
Summative
What Assessments are Used at
UAMS?
What Makes a Good
Summative Assessment?
Fair and practical
Based on appropriate criteria that
are shared with the learners
Valid
Reliable
Validity
Key to the interpretability and
relevance of scores
Do the scores have a plausible
meaning?
What are the implications of
interpreting the scores?
What is the utility of the scores?
What are the actual social
consequences?
Validity Evidence for Proposed
Interpretation
Content/Construct Evidence
Content matches the objectives &
teaching
Score is shown to be meaningful
Criterion/Predictor Evidence
Score relates to future outcomes
Reliability
Assessment data must be
reproducible
Written tests
•
•
Internal consistency
Cronbach’s alpha or KR-20 (values
range 0-1)
Test-retest
Correlation ranging from 0-1
Reliability (cont.)
Rater-based assessments
Interrater consistency or agreement
(generally correlation coefficient or
intraclass correlation)
OSCE & performance assessments
Generalizability Theory
Improve Validity
Match assessment with objectives
Increase the sample of objectives
and content on the assessment
Review blueprint (or produce one)
Use test methods appropriate to
objectives
Ensure test security
Improve Reliability
Clear questions
Appropriate time allotted
Simple, clear and unambiguous
instructions
High quality scoring
Increase number of observations or
questions
Norm or Criterion Referenced
Norm-referenced (relative standard)
Use the results of the assessment to set the
standards (e.g. proportion to receive each
grade); performance judged by comparison
with others
Criterion-referenced (absolute
standard)
Learners achieve some minimal standard of
competency
Standard Setting Procedures
Judgments based on
Holistic impressions of the exam or
item pool
Content of individual test items
Examinee’s test performance
Anghoff Standard Setting
Procedure
Judges are instructed to think of a group
of “minimally acceptable” persons
For each item estimate the proportion of
this group that will answer item correctly
Sum up proportions to get the minimum
passing score for a single judge
Average across judges
Assessment Methods
ACGME Toolbox
Additional methods
Other written assessments
•
•
•
•
Essay
Short answer and computation
Patient management problems
Modified essay questions
Create a game
Self-assessment
Matching Assessment Method
and Objectives
Examine ACGME matrix
Alternative Testing Modes
Take-home tests
Open-book tests
Group exams
Paired testing
Reducing Test Anxiety
Make the first exam relatively
easier
Give more than one exam
Give advice on how to study
Encourage study groups
Give a diagnostic test early
Reducing Test Anxiety
Before a test explain format
Provide sample items
Provide a pool of final items with
syllabus
Technology and Assessment
Computer based testing
Use of images and video clips
Computer Adaptive Testing (CAT)
Take a Test
Multiple Choice Questions:
Writing Tips
http://www.nbme.org/about/itemwriting.asp
Simple suggestions:
Design for one best answer
Place hand over stem. Can you
guess the answer?
Use a template
Attend to Graveyard of Test Items
Template Examples
A (patient description) is unable to
(functional disability). Which of the
following is most likely to have
been injured?
Following (procedure), a (patient
description) develops (symptoms
and signs). Laboratory findings
show (findings). Which of the
following is the most likely cause?
Graveyard
B-type: matching heading with
words or phrases (can use heading
more than once)
D-type: complex matching—two
step process
K-type: stem with four options and
choices are
A=1,2,3 only B=1,3 only C=2,4 only
D=4 only, E=all are correct
Portfolios
Portfolios
Definition
A purposeful collection of student
work that exhibits to the student
(and/or other) the student’s efforts,
progress, or achievement in (a) given
area(s).
This collection must include student
participation in
Selection of portfolio content
The criteria for selection
The criteria for judging merit
Evidence of student
reflection
Portfolios
Learning
Showing progress
Showing effort
Showing
remediation
Assessment
Showing
competence
Presentations
Logs
Case Records
Evaluations
Criteria
______
______
______
Criteria
Portfolio
Using Portfolios the Learner
Engages in Assessment
Making choices
Critically self-examining
Explaining what they are doing
Well-structured Portfolio
Menu
Selection criteria
Scoring rubric
Benchmarks
Trained raters
Determine Reliability and Validity
Reliability: Generalizability theory
Found needed 6 entries with 2 raters
Validity: number of studies
Resident performance across years
Resident performance in areas
Resident perception
Resident performance when program made
an intervention
Correlation of portfolio performance and
other areas
10-Step Approach
1.
2.
3.
4.
5.
Define the purpose
Determine competencies to be
assessed
Select portfolio materials
Develop an evaluation system
Select and train examiners
10-Step Approach
6.
7.
8.
9.
10.
Plan the examination process
Orient learners
Develop guidelines for decisions
Establish reliability and validity
evidence
Design evaluation procedures