Transcript 'The Design and Use of Multiple Choice Tests'

"The Design and Use of
Multiple Choice Tests"
Multiple Choice Exams
“ Written properly, multiple choice
exams correlate strongly with
assessments by descriptive tests ”
Brown, Robert, “Multiple Choice Versus Descriptive Tests”,
Frontiers in Education Conference, Reno NC 2001.
Use of Multiple Choice Exams
Not limited to Factual Knowledge

Making inferences

Application of models

Evaluation
Multiple Choice vs. Descriptive
Examinations
Descriptive
Questions
Easy
Multiple
Choice
Grading Task
Difficult
Easy
Grading Time
Long
Short
Grade Consistency
Varies
High
Setting the exam
Difficult
Structure of a
Multiple Choice Question
Stem
Answers
Formulating Good
Multiple Choice Questions

Wording the stem so that there is a single
correct answer

Finding plausible wrong answers

Sensitivity to ESL students
-
Negative voice in the stem
Conditional “if… would” structures
Cultural contexts for questions
Stems
Statements
“A river that flows through a desert is called..”
Questions
“ Which of the following descriptions best fits
the character of Alsatian wines?”
Negative voice
“ Which of the following grapes is not used to
make Amarone?”
Stems
Conditional Voice
“ If you want a VQA Niagara red wine that
was very fruity and required only a short
ageing period you would select: ”
Answer Options
Single correct answer options
 Collective answer options:
“ (a) and (c) are both correct ”
“ all of the above are correct ”
“ none of the above are correct ”

Plausible Answer Strategies

Good questions contain several plausible
answer options

Grey area between designing “tricky”
questions and providing answer options
that test knowledge thoroughly.

Inadvertently giving away the answers
Giving away the answer…
A verbal question:
The forbidden fruit that Eve offered Adam
was an:
(a) apple
(b) pear
(c) banana
(d) melon
(e) carrot
Giving away the answer
i.e. a numerical question:
It costs a vendor $1.00 each time s/he
sells a hot dog to a customer. If the price
of the hot dog is $3.00, what is the
contribution margin percentage?
(a) 17.5%
(c) 47.5%
(e) 82.0%
(b) 22.5%
(d) 66.7%
“Plausible” Answer Strategy
It costs a vendor $1.00 each time s/he sells
a hot dog to a customer. If the price of
the hot dog is $3.00, what is the
contribution margin percentage?
(a) 300%
(c) 67.7%
(e) 33.3%
(b) 200%
(d) 50%
Plausible Answer Strategies
Use Nonsense Words :
Colio Wines in Ontario produce an interesting
product classified as Vin de Curé in much the
same way as Amarone is made in Veneto.
The Italian word describing the drying process
used to desiccate the grapes is:
a) Ripasso
c) Raisinato
e) Prunello
b) Appasimento
d) Pommace
Plausible Answer Strategies
Use related, but different terms….
The word that refers to the addition of
sugar to a finished wine is:
(a)
(b)
(c)
(d)
(e)
Amelioration
Chaptalization
Edulcoration
Elaboration
Oxidation
Adding Uncertainty..
The term "fortified wine" means:
a) the bottle had been reinforced
b) the wine has had tannins and acid
added
c) the wine has had sugar added
d) the wine has had SO2 added
e) none of the foregoing
Adding difficulty..
Choose the best answer.
Rootham’s, a well-known local manufacturer of
brandied fruit preserves sells their product
through gourmet food shops and at craft fairs
throughout Canada. Their recipes use genuine
brandy and all natural ingredients. They are
able to sell these legally because:
(a) the alcohol used is produced by fermentation
(b) the product is made with ethyl alcohol
(c) the alcohol used is purchased from the LCBO
(d) the alcohol concentration is less than 1.2%
(e) the label says “made with genuine brandy”
Making Multiple Choice Questions
Interesting…
A gracious, but parsimonious hostess wants to
receive guests in the private dining room of your
restaurant with a glass of champagne just to
tantalise the taste buds and set the mood for the
evening.
1.You suggest that if the wine is being poured at
the bar, she should consider serving a Canadian
sparkling wine that is very similar to Champagne
in quality but considerably less expensive. Such
a wine would be a:
2. The cheapest sparkling wine would be
obvious to sophisticated guests because
they would know that in the best sparkling
wines…
3. If the gracious hostess insists on having
“French” Champagne and you are left to
rationalise costs, the most cost effective
solution is to serve Champagne from..
4. Occasionally, a patron asks you to do
something that will distinguish their event
from others. One client wants to obtain a
large bottle of Champagne to celebrate a
particular event. Which is the largest
bottle of champagne on the list below?
5. If someone wanted to serve a "glass of
bubbly" during a reception, which of the
following would be the most inappropriate
type of wine to select as the pre-prandial
(before dinner) libation?
Evaluating Multiple Choice
Questions
Difficulty Factor = R/N
R = # of times the correct answer is
selected
N = # of people taking the exam
Too Easy
Too Difficult
0
0.5
+1
Difficulty Factor
Values greater than 0.8:



The question is too easy
Stem wording gives away the answer
Not enough plausible incorrect answer options
Values less than 0.2:





Guessing effect can be a factor
The question is defective
Material wasn’t covered
Stem wording is unclear, confusing
Answer options need work
Evaluating Multiple Choice
Questions
Discrimination Index =
Proportion of capable students that select an
answer option – proportion of weak students
that select the answer option
i.e. D = Ftop quarter – Fbottom quarter
Reverse
Discrimination
-1
0
+1
Good
Discrimination
Discrimination Index
For the Correct Answer Option
Values ~ 0.5 give good discrimination
< 0.2 consider dropping
negative… drop the question
For Incorrect Answer Options
Values should be negative.
Knowledge or Successful
Guessing?
Multiple Choice Exam Strategies

Improve odds by eliminating 1 or more
infeasible or unlikely answer options
Descriptive Exam Strategies
Brain dumping
 Part marks
 Consideration for perfect answers to
questions that were not asked

Possibility of a “Random Pass”
Depends on the number
of answer options per question
and the number of questions!
Number of
Questions
1
2
4
6
10
20
50
Percent Pass (≥50%) by Chance
2 choices
50
75
69
66
62
59
56
3 choices
33
56
41
32
21
9.2
1
4 choices
25
44
26
17
8
1.4
.01
5 choices
20
36
18
10
3
.3
.0004
from Brown, 2001
Adjustment for Guessing
Negative Marking…

Elimination strategy reduces odds of wrong
answer penalty

Subtracting a percentage of the number of
wrong answers obtained from the final grade

Give a grade of 4 for a correct answer and a
score of -1 for a wrong answer on a 4 choice
question
Negative Marking..

A score of less than zero is possible

Students hate negative marking

Negative marking is not practised in
descriptive examinations

A poor substitute for a test that is
too short with too few answer
options
Teaching With
Multiple Choice Questions

Prepare lectures incorporating all material
covered in the questions to be used

After each lecture, check coverage by
reviewing questions

Provide students with multiple choice
questions for self-evaluation at the end of
readings
Teaching With
Multiple Choice Examinations
Use stem questions to teach additional
or ancillary terms not covered in the
course:
i.e. …. For post-prandial (after dinner)
drinks you want to stock a supply of
fortified wines and brandy. The term
"fortified wine" means….
Provide Students the Stem
Questions Prior to the Exam!
Exam questions should cover all the
course material being tested
 Provides more detail than a course outline
 Students focus their study effort on the
topics they understand the least
 Increases learning
 Reduces stress

Thanks for your interest in my topic!
I look forward to your questions and ideas.
J.E. (Joe) Barth
School of Hospitality and Tourism Management
University of Guelph, Ontario CANADA