Transcript MCQs 3 - Aligning with Course Objectives

MCQ’s 3:
Aligning with Course
Objectives
MCQ Check-list
This presentation will focus on:
 Why we write them
 What type is best
 What flaws are to be avoided
 What content to test
 How to construct it

Why do we test?

Formative assessment



Summative assessment



Communicate to the students what material is
important
Identify areas of deficiency in need of remediation or
further learning
Determine final grades
Make promotion decisions
Both


Motivate students to study
Identify areas where the course/curriculum is weak
What Should Be Tested?



Exam content should match course
objectives
Important topics should be weighted more
heavily than less important topics
The testing time devoted to each topic
should reflect the relative importance of
the topics
What Should Be Tested?


Sampling - Performance on the sample
provides a basis for estimating
achievement in the broader domain that is
actually of interest
Questions sample:


TOPICS – eg MCC presentations
SKILLS – eg Course Objectives – determining
diagnosis, next step in management, etc.

Term 4A, Jan 2012
Course Blueprint – ‘Alignment’
Types of MCQ’s
True/False family
 Assess recall of isolated facts



Stems must be clear and unambiguous
Options must be absolutely true or false or
examinee must decide how true options are
Generally not recommended – usually
have subtle flaws
Types of MCQ’s: True/False
Which of the following statements is true regarding
contraception?
A)
B)
C)
D)
E)
oral contraceptives should not be initiated until six
weeks after therapeutic abortion to avoid the risk of
hypercoagulability
an intrauterine device can provide good contraceptive
protection 5 years after insertion
oral contraceptives should not be prescribed to
individuals under the age of 15 without parental
consent
lesbian women do not require counseling about
contraception or sexually transmitted diseases
oral contraceptives should not be prescribed for women
over the age of 35
Types of MCQ’s
One-Best-Answer family (A type)
 Options may be diagrammed:
D C A
Least correct


E
B
Most correct
Correct answer is the most likely correct
Better application of knowledge, integration,
synthesis and judgment
Types of MCQ’s:
One-Best-Answer family
A 24 year old university student came to see you because
she is missing several days of school each month because
of severe dysmenorrhea unresponsive to the OCP and
NSAIDS. She also complains of deep dyspareunia and has
a tender uterosacral nodule on pelvic examination. Which
one of the following conditions is most likely responsible
for her symptoms?
A)
B)
C)
D)
Chronic pelvic inflammatory disease
Endometriosis
Adenomyosis
Fibroids
MCQ Check-list
Why we write them
 What type is best
 What flaws are to be avoided
 What content to test
 How to construct it

What to test?
Items can be classified by cognitive processes
required to answer the question:
 Recall (Memory)– tests knowledge of isolated
facts
 Interpretation (Comprehension)– review
information to reach conclusion, eg diagnosis
 Problem solving (Reasoning) – situation requires
action, eg next step in management

Process depends on background of
trainee
What to test?
Simpler classification based on task:
 Application of knowledge item **




Reach conclusions
Make predictions
Select course of action
Recall item

Test knowledge of isolated facts
What to test?

Test application of knowledge using
clinical vignettes



Adds ‘face-validity’
Usually selects more important, less trivial
Identifies student who’ve memorized but do
not understand
BUT experience in patient care should not
be necessary
What to test - summary




Focus item on key concepts and principles that
are essential information (without access to
references) for all examinees to understand
Test material that is relevant to learning in preclinical courses and beyond
Avoid items that only require recall of isolated
facts
Avoid esoteric or interesting topics that are not
essential
What to test - summary



Test application of knowledge using
clinical vignettes to pose medical decisions
in patient care situations
Focus items on common or potentially
catastrophic problems
Pose clinical decisions that would be
expected of a successful examinee

ie Avoid clinical situations that would be
handled by a (sub)specialist
Writing the Questions



Construct stem –include all necessary
information to arrive at the right answer
Choose lead-in – pose a clear question
Choose distractors
Stem*




Usually a clinical scenario
Clear, unambiguous
Should be long relative to options
Includes all pertinent information




Patient’s age/gender
Clinical setting
Complaint
Other important info – history, physical
findings, test results
Lead-in*




Follows stem
In the form of a question
Should relate to the stem
Should be answerable without looking at
options


“Cover the Options test”
Should clearly communicate the learning
objective
Distractors


Each should be selected by some,
therefore all plausible, none obviously
incorrect
3 or 4 choices


Common misconceptions
Faulty reasoning
Distractors






Homogeneous in content to the correct answer
Should be in same category as correct answer–
diagnosis, treatment, diagnostic test
Clearly incorrect or inferior to the correct answer
Plausible and attractive to the uninformed
Similar to the correct answer in construction and
length
Grammatically consistent and logically
compatible with the stem
General Guidelines for Item
Construction



Include as much of the item as possible in the
stems: Long stems, short options
Avoid negatively phrased items (except or not in
the lead-in). If you must, use only with short
options
Avoid writing questions of the form:


Which of the following statements is correct?
Each of the following statements is correct EXCEPT


Unfocussed
Heterogeneous options
“Top-down” versus “Bottom-up”
Top-Down
 Knowledge or recall question
 Cite a disease and then ask what patient
findings are expected
 Structured like textbooks
Clinically backward
“Top-down” versus “Bottom-up”
Bottom-up
 Application of knowledge
 Gives findings and asks examinee to
indicate the disease
 Examinees need to be able to synthesize
information from several pages of a
textbook (or notes) to answer these
questions
Clinically realistic
Basic Rules for MCQ’s (A type)*





Focus item on important concept
Item should assess application of
knowledge, not recall of an isolated fact
Stem must pose clear question – ‘Cover
the options’ test
All distractors should be homogeneous
Avoid technical item flaws
Reference
Constructing Written Test Questions For the
Basic and Clinical Sciences

Case & Swanson, NBME 1996
Clinical Presentation
A 28 year old female presents to the ER with a 2 hour
history of severe right sided abdominal pain and
vomiting.
Clinical Presentation
A 28 year old female presents to the ER with a 2 hour
history of severe right sided abdominal pain and
vomiting.
Clinical presentation
A 28 year old female presents to the ER with a 2 hour
history of severe right sided abdominal pain and
vomiting. This was preceded by a 1 week history of
urinary frequency and urgency. On physical
examination, VSS are stable and she is afebrile.
Abdomen is non-tender. She is noted to have mild
Right CVA tenderness.
Clinical Presentation
1) A 28 year old female presents to the ER with a 2 hour
history of severe right sided abdominal pain and vomiting.
This was preceded by a 1 week history of urinary
frequency and urgency. On physical examination, VSS are
stable and she is afebrile. Abdomen is non-tender. She is
noted to have mild Right CVA tenderness. Which of the
following investigations would be the next step to make a
diagnosis?
a) abdominal X-ray
b) abdo/pelvic USS
c) urinalysis
d) CBC
Presentation
2) A 28 year old female presents to the ER with a 2
hour history of severe right sided abdominal pain and
vomiting. This was preceded by a 1 week history of
urinary frequency and urgency. On physical
examination, VSS are stable and she is afebrile.
Abdomen is non-tender. She is noted to have mild
Right CVA tenderness. Urinalysis reveals +++ blood,
+ WBC and nitrites neg. CBC is normal. Which of the
following is the most likely diagnosis?
a) appendicitis
b) pyelonephritis
c) nephrolithiasis
d) cystitis
Presentation
3) A 28 year old female presents to the ER with a 2 hour
history of severe right sided abdominal pain, preceded by
urinary frequency and urgency. VSS are stable and she is
afebrile. She is noted to have mild Right CVA tenderness.
Urinalysis reveals +++ blood, + WBC, nitrites neg. You
suspect nephrolithiasis. Which of the following
investigations is the best choice to confirm the diagnosis?
a) Abdominal and pelvic USS
b) abdominal x-ray
c) CT scan
d) IVP
Presentation
4) A 28 year old female presents to the ER with a 2
hour history of severe ride sided abdominal pain with
R CVA tenderness. A CT scan confirms the presence
of a R UVJ stone measuring approximately 3 mm with
complete ureteric obstruction. There are no signs of
Urinary Tract infection. You provide analgesia. Which
of the following is the next step in managing this
problem?
a) Hydration
b) Percutaneous nephrostomy
c) Cystoscopy and basket retrieval of stone
d) Lithotripsy
Presentation
5) A 28 year old female has recently recovered from her
third bout of renal colic. She is otherwise healthy.
Which of the following investigations would be most
helpful in finding the cause of her recurrent
nephrolithiasis?
a) IVP
b) CT scan
c) serum Calcium
d) 24 hour urine calcium
Presentation
6) A 28 year old female has recently recovered from her
third bout of renal colic. She is otherwise healthy.
Investigations reveal no evidence of
hyperparathyroidism, urinary tract anomaly or other
underlying condition. Which of the following would be
most helpful in preventing this from recurring?
a) hydrochlorothiazide
b) Flomax
c) cranberry juice
d) 8 glasses of water daily
Basic Rules for MCQ’s (A type)*





Focus item on important concept
Item should assess application of
knowledge, not recall of an isolated fact
Stem must pose clear question – ‘Cover
the options’ test
All distractors should be homogeneous
Avoid technical item flaws
One last step…

Get a colleague to read over your
questions




Detects more than one right answer
Discovers the ‘what am I thinking?’ question
Identifies not enough information in the stem
…