Transcript Document

Critical Thinking in
Medical Education:
Assessing What We Mean
and What We Know
Ed Krupat, PhD
Director, Center for Evaluation
Medical Education Grand Rounds
Dec. 5, 2008
Components of Talks
that I Have Liked
1. A bit of context or framing
2. A touch of theoretical/conceptual
background
3. A good portion of research and data
4. A bounty of concern for practical
application
5. A strong dose of provocative discussion
Context: The Origins
of These Efforts
• Derek Bok--to--Richard Hersh
– Evaluation of programs
• Confusion-to-clarity-to-confusion-to-???
• Why is critical thinking so important
– In life
– In medicine
• Desire to be data-driven
Conceptual Analysis:
Just What Is Critical
Thinking??
• Overlapping concepts
– Analytic reasoning
– Problem solving
– Decision making
– Clinical/diagnostic reasoning/judgement
– Habits of mind
– Meta-cognition
– Adaptive expertise
Some Definitions:
Critical thinking is…
• ...the intellectually disciplined process of
actively and skillfully conceptualizing,
applying, analyzing, synthesizing, and/or
evaluating information gathered from, or
generated by, observation, experience,
reflection, reasoning, or communication as a
guide to belief and action. (Scriven, 1996)
• …the art of thinking about your thinking while
you are thinking in order to make your
thinking better: more clear, more accurate,
and more defensible. (Paul et al, 1989)
Bloom’s Taxonomy
Eval.
Synthesis
Analysis
Application
Comprehension
Knowledge
Research & Data
• Two projects being conducted
simultaneously
1.Qualitative: Do physicians agree about just
what critical thinking is?
2.Quantitative: Can we assess critical
thinking among medical students?
What is Critical Thinking:
The Responses of Physicians
• Survey of practicing MD faculty at 5
medical schools
– HMS, UCSF, Case Western, U Mass,
Baylor (total n=73)
– Convenience sample
Two Tasks
1. Define critical thinking
2. Think of a clinical scenario in which critical
thinking was important
•
•
•
•
describe it
state what a good critical thinker would do or say
in that situation
state what a poor critical thinker would do or say
state how the outcome would differ if one or the
other would have been the physician involved
Coding of the Definitions. 1
• Definition specifically included:
– Collection of information/data: 27%
– Making sense of information: 96%
– Utilization for decision making: 74%
– Utilization for action: 14%
– Specific ties to medical context: 43%
– Necessity of building upon knowledge
base: 27%
Coding of the Definitions. 2
• Critical thinking characterized as
– A process
– A skill or ability
– A disposition
Critical Thinking as a Process
• …a process of reflective reasoning that uses
objective evidence, a deliberate weighing of
options and alternatives, and clinical
judgement and experience to guide decision
making.
• …the process by which one is able to
rationally acknowledge different choices,
processes and outcomes in the clinical
encounter.
• a process in which problems are being
analyzed from different angles and connected
to pre-existing knowledge before any
conclusions are being drawn
44% of all responses
Critical Thinking as a
Skill or Ability
• …the ability to think through a problem using
reasoning. Also the ability to judge the
credibility of sources.
• …the ability to rigorously weigh the validity of
evidence and then to effectively synthesize
this evidence to reach a clinical decision.
• …the ability to effectively problem solve using
known data or under conditions of
uncertainty.
50% of all responses
Critical Thinking as a Disposition
• …careful attention to what you know, vigilance for
what you do not, and the courage to question both of
the above categories.
• …thinking about an topic, issue, or challenge in a
way that sets aside my immediate ’gut’ response,so
that I can be open and reflective to other possible
ways of viewing the challenge…
• …thinking deeply, keenly, flexibly, openly, reflectively,
with an awareness of self and others, with attention
to what is known and unknown, and with humility.
10% of all responses
Breakdown by Specialty
0.9
0.8
0.7
0.6
Medic ine
0.5
0.4
S urgery
0.3
P s yc hiatry
0.2
O ther
0.1
0
A bility
Dis pos ition
P roc es s
Clinical Situations:
An Interim Report
• Looking for
– What situations, choices, challenges differentiate
critical and non-critical thinkers
– What is it that critical thinkers do or say
– What is it that non-critical thinkers do or say
– What outcomes differ when critical thinking is or is
not in evidence
• Bottom line:
– What are the key differences in the thoughts,
words, and actions of critical thinkers
Clinical Contexts and Tasks
• Not surprising
– Diagnosis and treatment
• Surprising
– Collaboration and interpersonal issues
• How to protect patient’s rights and autonomy
• How to deal with difficult patient requests
• How to provide patient with appropriate options
The Differences
• Critical thinkers:
–
–
–
–
–
–
–
–
Do more data gathering
Avoid premature conclusions
See inconsistencies in information
Utilize knowledge more extensively and explicitly
to make decisions
Are aware of limitations and doubts
Monitor and evaluate their own decisions
Involve patients more fully
Provide patients with options
A Few Conclusions,
Many Questions
• Physicians are not necessarily all speaking
the same language when they discuss critical
thinking with one another
• Is critical thinking a skill, a process, or a way
of looking at the world?
– Are skill and disposition both necessary?
– Do you teach/encourage/foster skills in the same
way as dispositions???
– Do you assess skills in the same way that you
assess dispositions?
• Where does building upon a knowledge base
come in?
How do we assess
critical thinking?
• Collegiate Learning Assessment (CLA)
–
–
–
–
Used in over 200 colleges and universities
Assesses higher order outcomes
Students work on 90 minute performance tasks
System has been devised to generate reliable
scores
• Question: Is this a valid measure of critical
thinking for medical students?
Catfish: One of the
Performance Tasks Used
• Grotesquely mutated catfish has been found
in the local lake that supplies the town’s water
• You will serve on the mayor’s advisory panel
• You are provided with 6 documents to read
– Newspaper article
– Editorial by environmental activist
– Radio interview with a biologist from a nearby
college
– State report on water testing from lake
– Area map
– Journal article about similar discoveries
Task
• Open-ended written questions ask
students to:
– Identify main hypotheses to explain
phenomenon
– Identify strengths and weakness of each
– State and defend most likely explanation
– Suggest course of action
Scoring
• Evaluation of evidence
– What is relevant, what is valuable
• Analysis and synthesis of evidence
– Connections, inconsistencies, flaws in reasoning
• Conclusion drawing
– Acknowledging alternatives and options
• Presentation of arguments
– Concise, evidence-based, logically structured
Research Design
• Recruit multiple schools
• Recruit and compare students at two (or
more) points in time
– Entering Year 1 students took two tasks
– Students at end of Year 3 took one task
• Look for associations between CLA scores
and other performance indicators
–
–
–
–
–
MCAT scores
Gender
Age
English as first language
Step scores
• Compare scores between first and third year
students
Analysis Problems
• Among year 1 students
– scores on the two tasks not as highly correlated as
would be expected
– Brain Boost task always given first
• Year 1 students spend consistently less time and do
consistently worse on Catfish (always second task) than
on Brain Boost
• Among Year 3 students--who only took one task, either
Brain Boost or Catfish--students take more time and
perform better than on Brain Boost
• Small n’s
Findings 1.
• Medical school students have good critical
thinking skills at baseline
– Across the 3 schools, year 1’s
• fell at the 87th percentile among graduating seniors
nationally
• were 1.25 standard deviations above the national mean
• Is that good news or, with selectivity of the
medical schools, should we expect to be
even higher??
Table 1. CLA performance by task, year, and school
First Year
Third Year
Task
School
Mean
SD
n
Mean
SD
n
Mean
Difference
t
All
HMS
1307
108
80
1332
143
54
25
1.14
All
S2
1291
109
29
1311
111
16
20
.59
All
S3
1311
85
72
1250
111
32
-60
-3.03**
BB
HMS
1335
109
80
1312
160
27
-23
-.85
BB
S2
1314
134
29
1307
124
8
-7
-.13
BB
S3
1328
106
71
1245
112
16
-83
-2.80**
** p<.01
Table 2. Correlations among tests
and demographic variables
CLA
Brian Boost
MCAT Total
CLA
BB
MCAT
Total
MCAT
Phys
MCAT
Bio
MCAT
Verbal
MCAT
Writing
Age
Gender
English
Step I
--
.77***
.22***
.17**
.15*
.20**
.25***
-.11
.03
.11
.11
--
.18**
.16*
.13
.13
.23***
-.17**
.08
.08
.06
--
.82***
.82***
.70***
.38***
-.01
.21**
.03
.61***
--
.57***
.31***
.32***
-.04
.24***
-.10
.48***
--
.37***
.25***
-.02
.16**
-.02
.59***
--
.31***
.04
.07
.19**
.42***
--
.05
.01
.01
.37***
--
.06
.21***
.01
--
.09
-.05
--
-.01
MCAT Phys
MCAT Bio
MCAT Verbal
MCAT Writing
Age
Gender
English
Step I
Significance codes: * p<.05, ** p<.01, *** p<.001
Note: Significance not adjusted for multiple significance tests
--
Findings to Come
• What is the correlation between CLA
scores and
– Step II CK
– Step II CS
– HMS Comprehensive Exam
What have we learned?
• Critical thinking has many dimensions, and
physicians may be talking past one another if
they don’t share meaning when they speak.
• Critical thinking can manifest itself in how to
deal with patients, not just in generating
diagnoses and treatment plans.
• Critical thinking involves being
– Sensitive
– Inclined
– Able
(Perkins & Ritchhart 2002)
What have we learned? 2
• Medical students enter school with solid
critical thinking skills
• Critical thinking is somewhat, but not greatly,
associated with other measured outcomes
• It is still unclear the extent to which medical
education fosters or hinders critical thinking
• We have more questions at the end of this
process than we did than when it began, but
the questions are
– more sophisticated
– more tied to possible action
Some Big Questions
• If we think that critical thinking is so important,
why don’t we explicitly design the curriculum
to assure that we “teach” it?
• If we wanted to do so, how and when would it
be taught?
• Should measures of critical thinking, as a skill
or disposition, be built into our admissions
screening process?
• Should critical thinking be a core competency
that is assessed throughout the curriculum?
as a graduation requirement? As a
requirement for licensure?