Transcript Examining Distractor Effectiveness In Modified Items for Students

Examining Distractor Effectiveness
In Modified Items for Students with Disabilities
Michael C. Rodriguez
University of Minnesota
Peter Beddow, Stephen Elliott, Ryan Kettler,
Andrew Roach
Taking Distractors Lightly
• Distractors tend to be fillers, to occupy the
other three or four options.
• Distractors are sometimes absurd or even
humorous.
• “It’s hard to write 3 or 4 good distractors!”
From Distractors to Attractors
• Attractors contribute to the overall quality of
the item and test.
• Attractors play a central role in determining
the difficulty of an item.
• Attractors are explicitly designed to inform us
about prevailing misconceptions and errors.
To be Consistent
We are compiling evidence from our own work
and the work of others regarding the utility of
being direct and explicit in our item writing, relying
on good item writing guidelines, elements of
Universal Design, concepts from Cognitive Load
Theory, and research based on language
complexity and accessibility for diverse students.
TAMI provides a systematic guide.
Effects of Item Modification
•
•
•
•
Item Difficulty
Item Discrimination
IRT Item location and Item Fit
Distractor (Attractor) Functioning
– Proportion selecting the distractor
– Point-biserial correlation (distractor-total r)
– DDF
– Qualitative Evidence
Option
A
B
C
D
N
10
161
36
25
%
4
69
16
11
Ptbs
-0.17
0.38
-0.23
-0.18
Option
A [B]
B [C]
C [D]
N
323
99
53
%
68
21
11
Ptbs
0.40
-0.25
-0.27
Option
A
B
C
D
N
10
161
36
25
%
4
69
16
11
Ptbs
-0.17
0.38
-0.23
-0.18
Option
A
B
C
D
N
139
24
39
31
%
60
10
17
13
Ptbs
0.37
-0.25
-0.21
-0.08
Option
A
B [C]
C [D]
N
323
88
67
%
68
19
14
Ptbs
0.35
-0.34
-0.10
Option
A
B
C
D
N
86
51
53
53
%
35
21
22
22
Ptbs
0.40
-0.28
-0.19
0.01
Option
A
B
C
N
246
107
112
%
53
23
24
Ptbs
0.47
-0.41
-0.14
Option
A
B
C
D
N
58
24
21
136
%
24
10
9
57
Ptbs
-0.27
-0.37
-0.23
0.59
Option
A [A]
B [C]
C [D]
N
46
35
377
%
10
8
82
Ptbs
-0.46
-0.36
0.61
Option
A
B
C
D
N
66
111
22
24
%
30
50
10
11
Ptbs
-0.22
0.49
-0.25
-0.24
Option
A [B]
B [C]
C [D]
N
349
67
57
%
74
14
12
Ptbs
0.61
-0.42
-0.38
Option
A
B
C
D
N
131
30
53
10
%
58
13
24
4
Ptbs
-0.31
-0.10
0.48
-0.07
Option
A [A]
B [B]
C [C]
N
123
85
265
%
26
18
56
Ptbs
-0.38
-0.23
0.51
Option
A
B
C
D
N
85
69
40
30
%
38
31
18
13
Ptbs
0.23
-0.26
0.21
-0.21
Option
A [A]
B [B]
C [C]
N
111
272
89
%
24
58
19
Ptbs
-0.03
-0.03
0.07
Attractors are Explicitly Intentional
•
•
•
•
What makes this option an attractor?
Is this an effective attractor?
How many attractors are needed?
How can we improve the effectiveness of the
item attractors?
• Does a focus on attractors rather than
distractors matter? Does it help us make
items and tests more widely accessible?