Transcript Taking the Guesswork out of Writing Better Multiple Choice Questions Presented to AMATYC February 2, 2015 by Patricia L.

Taking the Guesswork out of
Writing Better Multiple Choice Questions
Presented to AMATYC
February 2, 2015
by
Patricia L. Gregg, PhD (Patti)
Georgia Perimeter College
Multiple choice tests have their place …
… but it isn’t necessarily “everywhere”
Question “stem”
Should give a clear indication of what
the question is about
Don’t be too broad or generic
(e.g. Which of the following is true?)
Phrase in form of a question
Don’t use incomplete statements, fill in
the blank, etc.
Be consistent with grammar and
syntax
Don’t give clues about right answer
based on the wording of the stem
Use simple and positive constructions Avoid convoluted and negative
(e.g. Name the most common cause of …) constructions
(e.g. Name the least uncommon cause of …)
Use scenarios when possible but
Don’t include a lot of distracting or
irrelevant information (red herrings)
Answer options
Categories should be mutually
exclusive
Avoid overlapping categories
Have one clear right (or best) answer
Avoid “all of the above” “none of the
above” “both a and c” etc.
Be consistent with grammar and
syntax
Don’t give clues about right answer
based on the wording of the options
Should be same length and
construction
Don’t make the right answer longer or
shorter than other options, or phrased
differently
Should be equally plausible
No obvious wrong answers
Aren‘t plausible distractors “trick questions”?
Plausible distractors help to
• minimize lucky guesses
• ensure that you are truly measuring students’
learning, not their ability to recognize blatantly
irrelevant information
• diagnose key concepts that students may be
misunderstanding or confusing with one another
Poll #1
ITEM ANALYSIS can help you understand
• which of your incorrect answer options are
plausible “distractors”
• whether your test question effectively
“discriminates” between high-performing and
low-performing students
Can we test higher order thinking
with multiple choice questions?
Yes, but we need to
be vigilant!
Too often we claim to be
assessing application,
analysis, interpretation, etc.
but we are really assessing
recognition, computation or
calculation.
• Not all word problems test higher order skills!
• Examples of question items that could measure
higher order skills would require students to:
 Interpret data or graphs
 Recognize spurious
conclusions
 Distinguish between relevant
and irrelevant information
_____triangles have three sides of equal length.
a.
b.
c.
d.
Isosceles
Equilateral
Right
Scalene
Leading with the blank is an awkward
construction, especially for non-native English
speakers and others with reading challenges.
Which type of triangle has
all three sides of equal length?
a. Isosceles
b. Equilateral
c. Right
d. Scalene
Adapted from Cheung & Bucat, How Can We Construct Good Multiple Choice Items?
This question tests the same knowledge as the previous
example, but the construction is more familiar.
The mean of a distribution of test scores is the:
a. Most frequently occurring score
b. 50th percentile
c. Arithmetic average
d. Measure of score range
A university developed an aptitude test to use for
admission into its Honors Program. The test was
administered to a group of seven applicants who
obtained the following scores: 70, 72, 72, 80, 89, 94, 98.
The mean score on the aptitude test is:
a. 72
c. 82
b. 80
d. 90
from Zimmaro, Writing Good Multiple-Choice Exams
In order to correctly answer this question, the student must
still know the definition of “mean” but also must be able to
calculate the mean. However, the question does not assess
higher order skills such as application or interpretation.
We want students to see the relevance
of course content, but too many details in
a word problem end up testing reading
speed and comprehension, instead of
math skills!
Suppose you are a mathematics professor who wants to
determine whether or not your teaching of a unit on
probability has had a significant effect on your students.
You decide to analyze their scores from a test they took
before the instruction and their scores from another exam
taken after the instruction. Which of the following t-tests is
appropriate to use in this situation?
a. Dependent samples.
b. Heterogenous samples.
c. Homogenous samples.
d. Independent samples.
When analyzing your students’ pretest and
posttest scores to determine if your teaching has
had a significant effect, an appropriate statistic
to use is the t-test for:
a.
b.
c.
d.
Dependent samples.
Heterogenous samples.
Homogenous samples.
Independent samples.
Burton et. al., How to Prepare Better Multiple Choice Items: Guidelines for
University Faculty
cited in Duquesne University, Good, Better, Best: Multiple Choice Exam Construction
Test items with carefully constructed
answer options can help you diagnose
what concepts your students may be
confusing and/or misunderstanding.
The number of bacteria N(t) present in
a culture at time t hours is given by
N (t) = 2200(2)t. Find the number of
bacteria present when t = 4 hours.
(a) 35200
(b) 17600
(c) 70400
(d) 19360000
Test item courtesy of a GPC faculty member
The correct answer is (a).
A student would select (b) if they multiplied 2 x 4
instead of raising 2 to the 4th power.
A student would get (c) if they multiplied 2200 by
32, thinking that (2)4 is 32 rather than 16.
The student who selects (d) demonstrates no
understanding of the order of operation. In this
case, the student multiplies 2200 by 2 and then
raises it to 4. Since that is not one of the choices,
19360000 (4400 raised to 2) is the closest.
http://www.act.org/compass/sample/pdf/numerical.pdf
How could this item be stronger?
Do we really need 5 answer options?
The COMPASS publishers probably do this to
minimize the probability of guessing the correct
answer, BUT it is really hard to come up with 5
plausible alternatives, so if one is blatantly wrong,
we may as well just have 4 options.
Do all the answer options contribute to
diagnosis?
It seems unlikely that a student would
select 0% or 45% unless they had almost
no grasp of simple arithmetic, even if they
didn’t understand how to calculate
percentages.
Does the number of math courses taken
contribute to the strength of the question?
It may be an unnecessary complication for weaker
readers.
The skill being testing by this item is calculation of
percentages.
We want the student to demonstrate their
understanding that the second group is 60% of
25% not 60% of the total class; therefore the last
remaining group is 40% of 25%.
One possible re-write:
This year, 75% of the graduating class of
Harriet Tubman High School had been
accepted to college. Of the remaining class
members, 60% had applied to college but had
not been accepted. What percent of the
graduating class did not apply to college?
The calculation is exactly the same, but the red
herrings are removed.
http://www.act.org/compass/sample/pdf/numerical.pdf
Poll #2
How could this item be stronger?
Ensure that all answer options contribute to
diagnosis.
Eliminate extraneous information
e.g. exercise 3 times per week for 20 minutes.
IF this test item were intended to measure
higher order skills, we would keep the exercise
information and possibly add some additional
“what if” scenarios with different variables.
PAPERS
Burton, Sudweeks, Merrill & Wood (1991, 33 pp.) https://testing.byu.edu/handbooks/betteritems.pdf
Cheung & Bucat (2002, 10 pp.) http://www3.fed.cuhk.edu.hk/chemistry/files/constructMC.pdf
Zimmaro (2010, 41 pp.)
https://ctl.utexas.edu/sites/default/files/documents/Writing-Good-Multiple-Choice-Exams-04-28-10.pdf
INSTITUTIONAL WEBSITES
Brigham Young University:
https://testing.byu.edu/handbooks/14%20Rules%20for%20Writing%20Multiple-Choice%20Questions.pdf
Duquesne University: http://www.duq.edu/about/centers-and-institutes/center-for-teaching-excellence/teaching-andlearning/multiple-choice-exam-construction
Indiana University: http://www.iub.edu/~best/pdf_docs/better_tests.pdf
University of Texas: http://ctl.utexas.edu/teaching/assessment/planning/multiple-choice
Vanderbilt University: http://cft.vanderbilt.edu/guides-sub-pages/writing-good-multiple-choice-test-questions/
COMMERCIAL PUBLISHERS
Censeo Corporation, Guidelines for Writing Effective Tests: A Practical “Short Course” for Test Authors (2007, 9 pp).
Document not available online; appears company may no longer be in business; I have a PDF copy to share.
Magna Publications:
http://www.facultyfocus.com/articles/educational-assessment/seven-mistakes-avoid-writing-multiple-choice-questions/
Patricia (Patti) Gregg
Associate Director, Assessment and Reporting
Georgia Perimeter College
[email protected]
678-891-2571
@DrPattiG
www.linkedin.com/pub/patricia-gregg