Transcript Assessing Learners

The Teaching Center
Assessing Learners
The Teaching Center
Department of Pediatrics
UNC School of Medicine
The Teaching Center
Goals and Objectives
• Goal:
» Overview the process of assessment of
medical learners for the purpose of evaluation
• Objectives:
» Outline differences between feedback and
evaluation
» Demonstrate that assessment should be
matched with objectives
» Describe various assessment methods
» Review the “RIME” method of assessment
The Teaching Center
Evaluation vs. Feedback
• Feedback:
» Formative
» Ongoing during a learning experience
» Focused toward improvement
• Evaluation:
» Summative
» Completed at the end of a learning experience
» Descriptive of performance
» Often, for medical students, “the grade”
The Teaching Center
Underlying Tenet of
Assessment
• Assessment of the learner must match learning
goals and objectives.
» Plan for learner assessment after objectives
are written and before the start of the learning
encounter
» Multiple assessment tools are typically needed
to measure and document accomplishment of
every learning objective in a learning
encounter
The Teaching Center
Six Competencies to Consider in
Developing Learning Objectives and
Assessing Learners
•
•
•
•
•
•
Medical Knowledge
Patient Care
Communication Skills
Practice-based learning and improvement
Systems-based practice
Professionalism
Example Methods of
Assessment
The Teaching Center
1.
2.
3.
4.
5.
6.
Observation
Multiple choice and other written testing
Oral examination
Chart review, Procedure logs, Case logs
Portfolio Development
Standardized patient encounters and
Objective Structured Clinical Exams
(OSCEs)
The Teaching Center
1. Observation of the Learner
• Strengths:
» Observation forms can be used to assess any
competency if the learner has the opportunity
to demonstrate it in the observed encounters
» Commonly used and accepted in medical
education
• Weaknesses:
» Subjectivity
» Lack of reliability because of different observer
standards
» Only as good as the observer makes them
The Teaching Center
The Observation Form
• It is important to use observation forms that
provide behavioral anchors.
» Do not simply use ratings:
• Example: Professionalism that is
“outstanding”, “average”, or “below
average” makes for very subjective and
unreliable assessments
» Instead outline expected behaviors for each
level of achievement
• Example: Regarding professionalism
document specifics: maintains
confidentiality, demonstrates honesty,
maintains a professional appearance, etc.
• The more clearly the expected performance is
outlined, the less subjective the observation
evaluation form.
The Teaching Center
2. Multiple Choice/Written Tests
• Strengths:
» Efficient, inexpensive
» Assesses medical knowledge well
» Can assess some aspects of patient care
• Weaknesses:
» Poor assessment of communication skills,
professionalism, systems-based practice and
practice-based learning
» Sometimes reflects test-taking skill more than
content knowledge
The Teaching Center
3. Oral Examination
• Strengths:
» Can assess thought processes
» Depending on structure, can assess medical
knowledge, patient care, practice-based
learning, systems-based practice, patient care
and professionalism.
• Weaknesses:
» Does not assess communication skills with
patients
» Often not “real world” test of professionalism
» Can be subjectively evaluated if there are not
specifically defined expectations for responses
The Teaching Center
4. Chart Review, Procedure Logs
and Case Logs
• Strengths:
» Documents patient care and therefore can be
used for assessment of that.
» Can also be used to assess practiced-based
learning and improvement and systems-based
practice.
• Weaknesses:
» Does not assess communication skills with
patients or professionalism.
» Often does not assess the quality of care
delivered.
» Dependent on the documented findings, so
cannot be used to assess medical accuracy.
The Teaching Center
5. Portfolio Development
• Strengths:
» Allows for documentation of self-assessment
in any competency area
» Can allow for documentation of evaluation
from others as well
• Weaknesses:
» Only as good as what is put into it – very
dependent on learner effort
The Teaching Center
6. Standardized patient encounters
and OSCEs
• Strengths:
» Can be very effective at assessing all
competencies
» Particularly useful in assessment of
communication skills
• Weaknesses:
» Expensive and time-consuming
» Depend on learner buy-in and how realistic
the encounter is made
» Subjectivity is a possible concern so, again,
behavioral anchors and clear expectations are
important
The Teaching Center
Matching Objectives to Assessment
Examples
Objectives:
Assessment Method:
Communicate effectively  Standardized patient
encounter
Demonstrate knowledge  Multiple choice test
Perform an accurate
physical exam
 Observation
Utilize current evidence
 Portfolio containing
literature reviews
The Teaching Center
The RIME Method of Evaluation
•
•
•
•
Reporter
Interpreter
Manager
Educator
• The medical learner must go through these
steps sequentially. These steps can be used
to assess overall progress.
The Teaching Center
Tips for Grading Medical Students
• Consider the RIME method, with expectation
that the Honors medical student can Manage
data
• Combine opinions to come to group
consensus on the subjective pieces
• Clarify expectations and mark behaviors for
various levels of performance
• Use a variety of assessment strategies
The Teaching Center
Summary and Conclusion
• Learners should receive both feedback and
evaluation
• Match assessment strategies to learning
objectives
» consider each competency
• Use multiple assessment strategies
• Know the strengths and weaknesses of each
assessment strategy chosen