Transcript Selected Response Tests

From Classroom Assessment: What Teachers Need to
Know, (2004) Ch. 6, by W.J. Popham
What are selected response tests?
What are selected-response tests?
Binary choice items
Multiple binary choice items
Multiple choice items
Matching items
5 General Item-Writing
Commandments
5 general commandments
1. Thou shalt not provide opaque directions
regarding how to respond to your
assessment .
Opaque in this context means:
A) not transmitting heat
B) impenetrable to light
C) hard to understand
D) stupid, unintelligent
5 general commandments
1. Thou shalt not provide opaque directions
regarding how to respond to your
assessment .
Opaque in this context means:
A) not transmitting heat
B) impenetrable to light
C) hard to understand
D) stupid, unintelligent
5 general commandments (cont.)
2. Thou shalt not employ ________________
statements in your assessment items.
Example:
Indicate if the following statement is true or
false:
____Leaders of developing nations have tended
to distrust leaders of developed nations due to
their imperialistic tendencies.
5 general commandments (cont.
2. Thou shalt not employ __ambiguous__
statements in your assessment items.
5 general commandments (cont.)
3. Thou shalt not provide students with an
__________________clue regarding the
appropriate response.
A. context
B. deceptive
C. unintentional (i.e., not purposeful)
D. problematic
5 general commandments (cont.)
3. Thou shalt not provide students with an
__________________clue regarding the
appropriate response.
A. context
B. deceptive
C. unintentional (i.e., not purposeful)
D. problematic
5 general commandments (cont.)
4. Thou shalt not employ complex syntax in your
assessment items.
Mark the following directions as complex (C) or
simple (S) syntax, appropriate for a beginning level
ELL:
___ After reading the article and identifying the
cause and effect relationships, read the lists of
causes and effects below, and draw a line between
each main action and its resulting event.
5 general commandments (cont.)
4. Thou shalt not employ complex syntax in your
assessment items.
Mark the following directions as complex (C) or
simple (S) syntax, appropriate for a beginning level
ELL:
_C_ After reading the article and identifying the
cause and effect relationships, read the lists of
causes and effects below, and draw a line between
each main action and its resulting event.
5 general commandments (cont.)
5. Thou shalt not use vocabulary that is more
advanced than required .
Which of the following is a concise paraphrase of the
above statement.
a) Employ scintillating syntax.
b) Eschew obfuscating verbiage.
c) Imitate exhilarating phonology.
5 general commandments (cont.)
5. Thou shalt not use vocabulary that is more
advanced than required .
Which of the following is a concise paraphrase of the
above statement.
a) Employ scintillating syntax.
b) Eschew obfuscating verbiage.
c) Imitate exhilarating phonology.
Binary Choice Items
 Greatest strength - can cover large amount
of content in a brief time
 Greatest weakness – 50% chance of guessing
correctly.
(A large number of items reduces impact of
this probability)
Binary choice examples
(See handout) – Look over these
true/false reading questions and
identify how the binary choice
guidelines have been violated.
Binary Choice Guidelines
 Phrase items to elicit thoughtfulness
 Minimize negative statements
 Avoid double-concept items
 Balance response categories
 Maintain item length similarity
Multiple Binary-Choice items
A cluster of items is presented to
the students , requiring a binary
response to each item in the cluster.
Usually the items are related to an
initial statement or set of
statements.
Multiple Binary Choice Items
Research (Frisbie ‘92) shows these are
 Highly efficient for gathering data
 More reliable than other selected-response
items
 Able to measure same skills/abilities
 Slightly more difficult than multiple choice
 Perceived by students are more difficult
but more efficient
Multiple Binary-choice items
Other reasons to use them:
• If you use stimulus material that is new
to the students, the item can be more
intellectually demanding
• Students can respond to more M B-C
than M-C items in the same amount of
time – greater reliability
Multiple Binary-Choice Items
Read handout example – how
could it be improved?
Multiple Binary-Choice Guidelines
Separate item clusters with obvious
formatting
Make sure all items in a cluster are
linked in a meaningful way to the
stem (stimulus material)
Multiple Choice items
 Can measure knowledge OR ability to think
 One advantage - answers can differ in relative
correctness – students choose the best answer
 A weakness – students need only recognize
correct answer (not generate)
 Stem can be a direction question or an
incomplete statement – direction question
best for younger students
 4,5 alternative answers best
Multiple choice examples
(See handout)
Multiple Choice guidelines
 The stem should consist of a self-contained
question or problem
 Avoid negative statements
 Attend to alternative length (no unintended
clues!)
 Randomly assign correct answers to
alternative positions
 Never use “all of the above”/use “none of the
above” to increase difficulty IF it helps you
make the test-inference you want
Matching items
 Advantage: compact form/little space
on page – easy to tap into lots of
information efficiently
 Disadvantage: sometimes encourages
memorization of low-level factual
information
Matching item examples
(See handout)
Matching item guidelines
 Each list should be homogenous
 Use brief lists; place shorter words/phrases on
the right
 Employ more responses than premises
 Order the responses logically
 Make explicit the basis for matching and the #
of times a response can be used
 Place all premises and responses on a single
page
Conclusion
 Guidelines will help, but there’s no
substitute for lots of PRACTICE
 Run your tests by a colleague
 Variety in testing procedures = greater
validity
Why? A well-learned cognitive skill can be
displayed in many ways. We should be
promoting generalizable mastery
Discuss
 Why are selected-response tests often
criticized?
 Someone makes this statement:
“Good teachers shouldn’t ever use selectedresponse type questions on a test. They
don’t test higher order thinking skills.”
What would you say?