Transcript Using Test Item Analysis to Improve Students` Assessment (Song Gao)

Using Test Item Analysis
to Improve Students’
Assessment
Institutional Assessment starts with Classroom Assessment
Learning Objectives of This Session

1. Explain difficulty index and discrimination index

2. Calculate difficulty index and discrimination index

3. Identify ineffective distracters

4. Evaluate multiple-choice test items based on
analysis results

5. Apply table of specifications to improve content
validity
Purpose of Item Analysis

1. Ensure accurate measurement of knowledge or skill

2. Enhance student learning

3. Increase student engagement

4. Avoid demoralizing students

5. Increase confidence in drawing conclusions
1.
Outcome achievement
2.
Level of knowledge or skill mastery
3.
Teaching effectiveness
Components of a multiple-choice item
Test items used to measure the lowest level
of cognitive taxonomy are (stem)
a)
Analysis (distracter)
b)
Application (distracter)
c)
Knowledge (key)
d)
Comprehension (distracter)
The correct answer usually numbered
as 1
The wrong answers usually numbered
as 0
Two important indexes for Item Analysis
 Item

To tell how hard the item is
 Item

Difficulty Index
Discrimination Index
To tell how well the item to distinguish between
high ability and low ability students
Item Difficulty Index

Is defined as the percentage or proportion of test
takers who correctly answer the item.
P 
number of examinees
correctly answering
the item
number of examinees
For example, in a class of 30 students, if 20 students
get the item correct and 10 are incorrect, the item
difficulty index is 20/30 =0.67
 Range from 0 to 1

ITEM DIFFICULTY = NO. CORRECT / TOTAL
Students Item1 Item2 Item3 Item4 Item5
Robert
1
1
1
1
1
Millie
1
0
1
1
1
Dean
1
0
0
1
1
Shenan
1
1
0
1
1
Cuny
1
1
1
1
1
Corky
1
0
1
1
1
Randy
1
1
0
1
1
Jeanne
1
1
0
0
1
Iliana
1
1
1
0
1
Lindsey
0
0
0
0
1
Item p = 0.9 0.6
0.5
0.7
1.0
Optimal P Values for Items with Varying
Number of Options
Number of Options
Optimal Mean p Value
2
0.85
3
0.77
4
0.74
5
0.69
Special Assessment Situations and Item
Difficulty

Previously discussed item difficulty is most applicable to
norm-referenced tests

For criterion-referenced tests or classroom tests, it is
normal to have average p values as high as 0.9 because we
expect most students to be successful

If a test were developed to select the upper 25%, it would
be desirable to have items with p values that average 0.25

In summary, although a mean p of 0.5 is optimal, item
difficulty levels vary with purpose of a test.
Item Discrimination Index

Is defined as the difference of item difficulty between
those who succeeded (called upper group or highachievement group) and those who failed the test (called
lower group or low-achievement group)
D  PU  PL




D = discrimination index (range from -1 to 1)
PU = difficulty index in the upper group
PL = difficulty index in the lower group
For example, Pu=0.8, PL=0.3, D=0.8-0.3=0.5
Guidelines for Evaluating D Values
Discrimination Index
0.4 and larger
excellent
0.3-0.39
good
0.11-0.29
fair
0.00-0.11 (it is OK for classroom tests)
poor
Negative
miskeyed or major flaw
Student ID
1
2
3
4
5
6
7
8
9
10
Total
R
1
0
1
1
1
1
1
1
1
1
9
Q
1
0
1
1
1
1
1
1
1
1
9
G
1
0
1
1
1
1
1
1
1
1
9
I
1
0
1
1
1
1
1
1
1
1
9
B
1
0
1
1
1
1
1
1
0
1
8
F
1
0
1
1
1
0
1
1
1
1
8
E
1
0
0
1
1
1
1
1
1
1
8
T
1
0
0
1
1
1
1
1
1
1
8
S
1
0
0
1
1
1
1
1
1
1
8
C
1
0
1
1
1
1
1
1
1
0
8
K
1
0
1
1
1
1
1
0
0
1
7
M
1
0
0
1
0
1
1
1
1
1
7
O
1
0
0
1
1
1
0
1
1
1
7
A
1
0
0
1
1
1
1
1
0
1
7
D
0
1
1
1
0
0
1
1
0
1
6
N
0
1
0
1
0
1
1
0
1
1
6
H
0
1
0
1
1
1
0
0
1
1
6
L
0
1
1
1
1
1
0
0
0
0
5
J
0
1
1
1
0
0
1
0
0
0
4
P
0
1
0
1
0
0
0
0
0
0
2
Discrmination-index
Distracter Analysis

It allows you to examine how many students in the upper
group and the lower group selected each option on a
multiple-choice item

We expect distracters to be selected by more students in
the lower group than students in the upper group.

An effective distracter must be selected by some
students.
Distracter Analysis
Options
Item 1
A*
B
C
D
Number in the upper group
22
3
2
3
Number in the lower group
9
7
8
6
Distracter Analysis
Options
Item 2
A*
B
C
D
Number in the upper group
17
9
0
4
Number in the lower group
13
6
0
11
Building a Table of Specifications
1. Selecting content areas
2. Selecting learning outcomes to be tested
3. Determining the levels of objectives
4. Determining the question type
5. Determining the points for each question
6. Building a table
A Sample of Table of Specifications
Content Area
Learning Objectives
Level of
objective
Item Type
number
point
Item Analysis
1. Explaining item
difficulty and
discrimination index
2. Calculate P and D
Comprehension
Multiplechoice
2
2
Application
Constructed
2
4
3. Identify ineffective
distracters
Application
Multiplechoice
1
1
Apply table of
specifications
Application
Project
1
5
Preparing a
classroom test