Transcript Writing MCQ

Georgetown University
Center for New Designs in Learning & Scholarship and Office of
Faculty and Academic Affairs
Writing Multiple Choice Questions
Aviad Haramati, PhD
Peggy A. Weissinger, EdD
Georgetown University
School of Medicine
2/12/2013
CNDLS & OFAA
Georgetown University
1
Agenda
 Welcome & introductions
 Why do we assess?
 Common flaws in MCQ
 Guidelines for writing quality MCQ
 Let’s do it!
 Resources
2
Why do we assess?
3
Why do we assess?
Objectives
Activities
Assessment
4
Adapted from Felder & Brent ( 1991)
Why do we assess?
Objectives
Activities
What should my
students know? What
should they be able
to do?
Assessment
5
Adapted from Felder & Brent ( 1991)
Why do we assess?
Objectives
What type of activities
will promote and develop
this knowledge?
What type of activities
can help students
achieve the learning
objectives?
Activities
Assessment
6
Adapted from Felder & Brent ( 1991)
Why do we assess?
Objectives
How will I know that they
have achieved the
objectives?
Activities
Assessment
7
Adapted from Felder & Brent ( 1991)
Congruence!
Objectives
Activities
Assessment
8
Adapted from Felder & Brent ( 1991)
What am I testing?
Before creating an exam, ask yourself:
 What do the students have to know (or do)?
 What am I testing?
9
What am I testing?
Core Principles:
 Decide exactly what you want to test.
 Eliminate as many ambiguities as possible.
 Create distractors based firmly on what you want to
test.
 After it is over, use it as a learning opportunity!
– Give feedback for your distracters which explains
why they’re wrong.
– Give feedback for the correct answer explaining
why it’s right.
Adapted from http://hotpot.uvia.ca/howto/mcquestion.htm
TLISI - Georgetown University
10
Technical Flaws
Issues Related to Testwiseness
 Grammatical cues
TLISI - (NBME)
Georgetown
University Written Test Questions
Reference: Case & Swanson
Constructing
11
Grammatical Cues
A 60-year-old man is brought to the emergency
department by the police, who found him lying
unconscious on the sidewalk. After ascertaining that
the airway is open, the first step in management
should be intravenous administration of
a. examination of cerebrospinal fluid
b. glucose with vitamin B1 (thiamine)
c. CT scan of the head
d. phenytoin
e. diazepam
TLISI - (NBME)
Georgetown
University Written Test Questions, p. 19
12
Reference: Case & Swanson
Constructing
Technical Flaws
Issues Related to Testwiseness
 Grammatical cues
 Logical cues
TLISI - (NBME)
Georgetown
University Written Test Questions
Reference: Case & Swanson
Constructing
13
Logical Cues
Crime is
a. equally distributed among the social classes
b. overrepresented among the poor
c. overrepresented among the middle class and
rich
d. primarily an indication of psychosexual
maladjustment
e. Reaching a plateau of tolerability for the nation
a, b, c are collectively exhaustive
d, e may be partially correct but not same dimension as others
TLISI - (NBME)
Georgetown
University Written Test Questions, p. 20
14
Reference: Case & Swanson
Constructing
Technical Flaws
Issues Related to Testwiseness
 Grammatical cues
 Logical cues
 Absolute terms
 Long correct answer
TLISI - (NBME)
Georgetown
University Written Test Questions
Reference: Case & Swanson
Constructing
15
Long Correct Answer
Secondary gain is
a. synonymous with malingering
b. a frequent problem in obsessive-compulsive
disorder
c. a complication of a variety of illnesses and tends to
prolong many of them
d. never seen in organic brain damage.
TLISI - (NBME)
Georgetown
University Written Test Questions, p. 20
16
Reference: Case & Swanson
Constructing
Technical Flaws
Issues Related to Testwiseness
 Grammatical cues
 Logical cues
 Absolute terms
 Long correct answer
 Word repeats
TLISI - (NBME)
Georgetown
University Written Test Questions
Reference: Case & Swanson
Constructing
17
Word Repeats
A 58-year-old man with a history of heavy alcohol use
and previous psychiatric hospitalization is confused
and agitated. He speaks of experiencing the world as
unreal. This symptom is called
a. derealization
b. depersonalization
c. derailment
d. focal memory deficit
e. signal anxiety
18
Technical Flaws
Issues Related to Testwiseness
 Grammatical cues
 Logical cues
 Absolute terms
 Long correct answer
 Word repeats
 Convergence strategy
TLISI - (NBME)
Georgetown
University Written Test Questions
Reference: Case & Swanson
Constructing
19
Convergence Strategy
Local anesthetics are most effective in the
a. anionic form, acting from inside the nerve
membrane
b. cationic form, acting from inside the nerve
membrane
c. cationic form, acting from outside the nerve
membrane
d. uncharged form, acting from inside the nerve
membrane
e. uncharged form, acting from outside the nerve
membrane
20
Technical Flaws
Issues Related to Irrelevant Difficulty
 Options long, complicated, or double
TLISI - (NBME)
Georgetown
University Written Test Questions
Reference: Case & Swanson
Constructing
21
Long, Complicated Options
Peer review committees in HMOs may move to take action against a
physician’s credentials to care for participants of the HMO.
There is an associated requirement to assure that the physician
receives due process in the course of these activities. Due
process must include which of the following?
a. Proper notice, a tribunal empowered to make the
decision, a chance to confront witnesses against him/her,
and change to present evidence in defense.
b. Notice, an impartial forum, council, chance to hear and
confront evidence against him/her.
c. Reasonable and timely notice, impartial panel
empowered to make a decision, a chance to hear
evidence against himself/herself and to confront
witnesses, and the ability to present evidence in defense.
TLISI - (NBME)
Georgetown
University Written Test Questions, p. 23)
22
Reference: Case & Swanson
Constructing
Technical Flaws
Issues Related to Irrelevant Difficulty
 Options long, complicated, or double
 Numeric data not stated consistently
 Terms in options are vague
 Language in options not parallel
 Options in non-logical order
 Stems are tricky or unnecessarily complicated
TLISI - (NBME)
Georgetown
University Written Test Questions
Reference: Case & Swanson
Constructing
23
Tricky Stems
Arrange the parents of the following children with Down’s syndrome
in order of highest to lowest risk of recurrence. Assume that the
maternal age in all cases is 22 years and that a subsequent
pregnancy occurs within 5 years. The karyotypes of the
daughters are:
I: 46, XX, -14, +T (14q21q) pat
II: 46, XX, -14, +T (14q21q) de novo
III: 46, XX, -14, +T (14q21q) mat
IV: 46, XX, -21, +T (14q21q) pat
V: 47, XX, -21, +T (21q21q) (parents not karyotyped)
a.
b.
c.
d.
e.
III, IV, I, V, II
IV, III, V, I, II
III, I IV, V, II
IV, III, I, V, II
III, IV, I, II, V
TLISI - (NBME)
Georgetown
University Written Test Questions, p. 25.
24
Reference: Case & Swanson
Constructing
Technical Flaws
Issues Related to Irrelevant Difficulty
 Options long, complicated, or double
 Numeric data not stated consistently
 Terms in options are vague
 Language in options not parallel
 Options in non-logical order
 Stems are tricky or unnecessarily complicated
 Answer is “hinged” to a related item
TLISI - (NBME)
Georgetown
University Written Test Questions
Reference: Case & Swanson
Constructing
25
Item Shape
Appropriately Shaped Item
Long Stem
a.
b.
c.
d.
e.
Poorly Shaped Item
Short Stem
a.
b.
c.
d.
5/21/08
e.
TLISI - Georgetown University
26
When Writing Options...
…make sure that they are:
 Homogenous in content
 Incorrect or inferior to the correct answer
 Plausible and attractive to the uninformed
 Similar to the correct answer in construction
 Grammatically consistent and logically
compatible with the stem
TLISI - (NBME)
Georgetown
University Written Test Questions
Reference: Case & Swanson
Constructing
27
Your opinion, please!
Osteopenia is caused by which of the following?
A. Boron
B. Phosphorus
C. Magnesium
D. Strontium
28
Edited Item
A 55 year-old woman presents with osteopenia.
In reviewing her dietary profile, you suspect one
of the following may be deficient. Which is it?
A. Boron
B. Phosphorus
C. Magnesium
D. Strontium
29
Your opinion, please!
A 60 year-old male runner complains of being confused,
irritable and very lethargic. He has hypertension for
which he is taking 10 mg of Lisinopril, an angiotensin
converting enzyme inhibitor. He states that he drinks
~3L of water and eats “lots of fruits and vegetables” daily.
Which of the following is the likely factor producing his
symptoms?
A. Potassium depletion
B. Sodium depletion
C. Lisinopril resistance
D. Decreased renal blood flow
30
Edited Item
A 60 year-old male runner complains of being confused,
irritable and very lethargic. He has hypertension for
which he is taking 10 mg of Lisinopril, an angiotensin
converting enzyme inhibitor. He states that he drinks
~3L of water and eats “lots of fruits and vegetables” daily.
Which of the following is the likely factor producing his
symptoms?
A. Hypokalemia
B. Hyponatremia
C. Hypocalcemia
D. Hypomagnesemia
31
General Guidelines
 Bulk of question in the stem.
 Try “cover-the-options” test.
 Avoid superfluous information.
 Avoid “tricky” and overly complex items.
 Write grammatically consistent options.
 Avoid absolutes! (e.g. always, never, all)
And most important: Focus on important
concepts. Don’t waste time testing trivial facts.
TLISI - (NBME)
Georgetown
University Written Test Questions
Reference: Case & Swanson
Constructing
32
Let’s Do It!
 Write one multiple choice question.
TLISI - (NBME)
Georgetown
University Written Test Questions
Reference: Case & Swanson
Constructing
33
SAMPLE ITEM
A 25 year-old medical student who is using natural family
planning to chart her fertility, just started her Surgery
rotation. Since the start of the quarter, she has been
waking up at 4:00 a.m. every day and getting less sleep
than usual, as well as feeling generally stressed. During
her next menstrual cycle, which of the following phases is
most likely to be longer than normal for her? (MD)
A.
B.
C.
D.
Follicular Phase
Menstrual Phase
Ovulatory Phase
Secretory Phase
34
SAMPLE ITEM
When practicing the Billings Ovulation Method of
NFP, infertile days of the cycle include…
(MD)
A. Dry days after menses
B. Every day from the 4th day after the Peak Day
until the end of the cycle
C. Both A and B
D. All of the above
35
SAMPLE ITEM
Functional Residual Capacity or FRC
(AN)
A. Is best measured by following Peak Flows
B. Is the equilibrium volume when the tendency of
the lung to expand is balanced by the tendency
of the chest wall to retract after full inspiration
C. Is usually greater than the residual volume
D. Is maximized when the patient is in deep sleep
E. Cannot be maintained by active inspiration
36
Pulling it together!
Summary:
 Decide what you want to test.
 Well-written objectives make the job easier.
 Use guidelines for writing quality MCQ.
 Local resources here to help!
37
Contact information
Aviad Haramati
[email protected]
202-687-1021
Peg Weissinger
[email protected]
202-687-4556
CNDLS & OFAA
Georgetown University
38