Transcript Multiple –Choice Questions Dr.Einollahi

Multiple –Choice
Questions:
Increasing validity and
reliability
Dr.Einollahi
How can we
increase
the validity and reliability
of this
objective assessment tool
of general knowledge?
For trainees, they talk about:
General Exam -Taking Hints
(or test-taking strategies
Or test-taking skills )
And
For trainers,
We should talk about:
General Exam -Making tips
(or test-making strategies
Or test-making skills)
In their classic paper on
test-wiseness
Millman, Bishop and Ebel (1965)
defined
test-wiseness
as “a subject’s capacity to utilize the
characteristics and formats of
the test and/or the test taking
situation to receive a high score”.
Testwiseness
is independent of
the skills and/or knowledge
that the test is supposedly
measuring, therefore
differential test-wiseness
within a group of students is a
source of invalidity in test scores.
We have many times seen that:
Many questions have only two
plausible options,
accompanied by
"throwaway" options for
filler.
WE SHOULD AVOID THIS
TO HAPPEN!
We should avoid
Common Faults in
item making.
1-Each question in the test should measure an
important educational objective.
They should not be about trivial or
unimportant matters.
2-There should not be more than one concept
to be assessed in a single question.
3-There should not be repetition of the words
in the options.
The words that are going to be repeated
should be written in the stem of the test.
Not a good item designing!:
Compensatory mechanisms are most
disturbed in:
1-Septic shock
2-Hypovolemic shock
3-Dissociative shock
4-……
As you see, the word “shock” should be taken
to the stem.
4-The stem should be complete and meaningful.
5-All options and distractors should have the same
length,structure and should be about a single
subject.Avoid cueing effects.
6-The test should measure higher levels in cognitive
domain and not just “recall”.
7-The test should have just one correct answer or
just one best answer.
…and also it should not be a
“read my mind question!”
8-A good distractor is one that misleads the
lower level(weak) trainees and not the
higher(strong) ones.
9- Grammatical rules and consideration of
homogeneity(in terms of sentence
length,structure,subject,..) should be
followed for all of the options,
otherwise they may provide hints for the
trainee.e.g.
Not a well-designed item!
In portal hypertension we may
have an:
1- esophageal varicosis
2-pulmonary embolism
3-coronary thrombosis
4-sepsis
10-The place of the correct option should be
randomized.
11- Do not use any guiding cues.
These cues may be words such as:
Always,never,completely ,necessarily
,only,totally,must,…
(many students know that these options are
usually incorrect!)
Or words like:
Often,perhaps,may,generally,sometimes,..
(again many students know that these options
are usually correct!)
12-Do not design negative items as far as possible.
However since it is more difficult to create three
well-designed wrong options(distractors) than just
one,many examiners design negative questions.
In these cases , the negative word should be made bold,e.g.
All of the following statements
regarding…….. …….are correct EXCEPT:
13-Do not use options
like:
“All of the above” or
“None of the above”.
Perhaps you may wish to read!
Cueing effect
of
“all of the above”
on the reliability and validity
of
multiple-choice test items.
Eval Health Prof. 1998 Mar;21(1):120-33.
14-Do not use complex options
like:
If A and B are correct,choose A
If B and D are correct ,choose C,..
15-Do not use compound
sentences in the options ,
e.g.
A. This occurs in pernicious
anemia and also delayed
stomach emptying is observed in
these patients.
16-If you have designed a fill in
the blank item,do not put the
blank at the first part of the
sentence.Put it at the last part .
17-Do not use “doubly negative”
questions.e.g.
Which statement is not correct about
shock:
A- Myocardial Depressant Factor has
not any role in shock.
BCD18-Do not use “abbreviations” in the items.
19-Do not duplicate exactly the same sentences of the
textbook as far as possible.
20 -Do not give the information
relevant to one question , in
another test item.
21-Avoid imprecise terms.
22 – Each question should be
constructed independent of
another one.
Indeed lack of knowledge in one
item,should not cause failure to
answer to another questions.
In test-taking strategies ,it is said:
If you have to guess :
1-The length of choices can be a clue.
Choose the longest.
2-If two choices are similar, choose neither.
3-If two choices are opposites, choose one of them.
4-The most general alternative is usually the right
answer.
So ,we should avoid providing
these clues in our items!.
One of our main aims is
to reduce
the number of
lucky guesses
and
to increase
the number of
knowledge-based answers
‫باتشکر‬