Transcript Multiple Choice Questions: Do Yours Make the Cut?

Are You Testing What You Think You Are?
How to Successfully Evaluate Test Items
Mari-Wells Hedgpeth
[email protected]
Kathryn W. Smith
[email protected]
Session presented at the Conference for Higher Education Pedagogy
Blacksburg, VA
February 2015
Today’s Objectives
• Identify common violations of recommended item
writing guidelines.
• Recognize and categorize common technical
flaws in multiple choice questions.
• Develop qualitative strategies to systematically
review MCQs.
Purpose of Today’s Session
Background:
• Multiple Choice Questions that contain flaws
interfere with information gathered from exam
scores
• Classifying items as meeting recommended
guidelines or not will help participants minimize
testwiseness (guessing, cueing) from their own
exams
Well constructed items = Valid & Reliable exams
Definitions
Checklist Continued
The primary purpose of the stam is to remove the:
a.
b.
c.
d.
carm
denton
menice
stam bar
Why is this flawed?
The word in the stem (stam) is
repeated in the correct option.
Key words should not appear in
both the stem and the
distractors.
Source: adapted from Butler and Kreisle
Which of the following pairs has won the greatest number of
Abby awards?
a.
b.
c.
d.
Jones & Smith
Smith & Taylor
Smith & White
White & Allen
Why is this flawed?
Convergence. One alternative appears
in each answer option, (Smith appears
in three of the choices and White
appears in two) thus a testwise student
will be able to determine the correct
option by determining the answer
where the two dimensions converge.
Source: adapted from Butler and Kreisle
Stamation normally occurs when the
a.
b.
c.
d.
anstels rupture
immon falls and the denton is in place
octals rotates easily
ports pass over the carm
Source: adapted from Butler and Kreisle
Why is this flawed?
The longest, most specific option is the
correct one. Try to make all options
equal in length. All distractors should
relate to each other in kind/type and be
similar in grammar, length, and
complexity (eg., all diagnoses, all
treatments, all next steps in care, etc.)
Anstels with short grestels produce a foul scent:
a.
b.
c.
d.
never
frequently
usually
rarely
Why is this flawed?
All other options are vague, there is no
set of definitions for frequently, usually,
or rarely.
Clear language should be used.
Source: adapted from Butler and Kreisle
President Bush’s stamations are fecitious because
a. His anstels are underdeveloped.
b. His broctals have been removed.
c. His parents are crinkets.
Why is this flawed?
d. None of the above.
None of the above should be avoided.
No Answer
Source: adapted from Butler and Kreisle
This option tends to be used when the test
writer cannot create another plausible
distractor. The quality of the distractors not
the quantity should be kept in mind. These
options also minimize the effectiveness of
the item because a None of the Above
option measures the ability of the test taker
to pick out the incorrect answers instead of
his/her ability to identify the correct one;
decreases discrimination of the item and
overall test reliability.
Recent evidence shows that a majority of European royals
are addicted to bishah. Which of the following statements
about this evidence is correct?
a. Bishah addiction is hereditary.
b. A sample of European royals does not generalize to
commoners.
c. This evidence reflects addiction
rates in the general
Why is this flawed?
population which is estimated to be 60-70%.
All answers
areirune
True/False
options.
d. No conclusion can be drawn
because
samples
were
Why not reword to ask true/false
not taken.
No Answer
Source: adapted from Butler and Kreisle
questions? Problem: each option must
be addressed as true or false, but the
item is scored as a single-best answer
question. .
Frongals are
a. shaped as anstels if they are immediately separated from
their stams at the same time of ripening or if they are
gown in a vacuum.
Why is and
this flawed?
b. empowered to produce anstels
are capable of
Short stem that is not asking a question.
destroying dresels while their
stamsareare
The options
longfeeding
and seemon
unnecessarily complicated. Unfocused
crozers
stem – stem should be complete, contain
c. know to have remarkable as
regeneration
despite
much of the testabilities
item as possible
and
initial evidence of a weak require
defense
mechanism
under (or
the application
of knowledge
higher –are
analysis,
synthesis,
circumstances when (a) stams
covered
withevaluation)
cristles to
answer. A single problem should be
and/or (b) crozers attack at
night. so that it is meaningful by itself
presented
No Answer
Source: adapted from Butler and Kreisle
and its focus is on important concepts
(objectives) and not factual recall. Clinical
vignettes should be used.
Using your new knowledge
Use the Multiple-Choice Question Exam Review Checklist to
identify and record flawed items.
Disclaimer
• The practice slides have not been included in this
presentation because they are actual items that were used
on exams. We do not have permission to distribute these
items.
References
Azer, S. (2003). Assessment in problem-based learning course. Biochemistry and Molecular Biology
Education, 31(6), 428-424.
Case, S. & Swanson, D.B. (2001, 2002). Constructing Written Test Questions for the Basic and
Clinical Sciences (3 ed.). Philadelphia, PA: National Board of Medical Examiners.
Downing, S.M. (2005). The effects of violating standard item writing principles on tests and
students: the consequences of using flawed test items on achievement examinations
in medical education. Advances in Health Sciences Education 10, 133-143.
Haladyna,T.M., Downing, S.M. & Rodriguez, M.C. (2002). A review of multiple-choice itemwriting guidelines. Applied Measurement in Education 15: 309-334.
Hauge, L.S. (2002). UNPUBLLISHED Depts. of Surgery & Medical Education, University of
Michigan. Adapted from Haladyna, T.M., Downing, S.M. & Rodriguez, MC. (2002). A
review of multiple-choice item writing guidelines for classroom assessment. Applied
Measurement in Education, 15, 309-334.
Jozefowicz,R.F., Koeppen, B.M., Case, S., Gallbraith, R., Swanson, D. & Glew, R.H. (2002). The
quality of in-house medical school examinations. Academic Medicine, 77 (2), 156-161.
rd