Transcript Part #3 - Sullivan University
Part #3
© 2014 Rollant Concepts, Inc.
2
Assembling a Test
© 2014 Rollant Concepts, Inc.
#
Assembling a Test
•
First page = easier questions
•
Test should progress from easier to more difficult (Nitko, 2004)
© 2014 Rollant Concepts, Inc.
#
Assembling a Test
•
Unnecessary to group content topics
© 2014 Rollant Concepts, Inc.
#
Assembling a Test
•
Avoid the correct answer to be the same letter for more than 3 or 4 questions in a row
•
Vary the difficulty level within a content area
© 2014 Rollant Concepts, Inc.
#
Assembling a Test
•
Edit w/ having a colleague proofread for punctuation, grammar, reading level, ‘makes sense’, clarity and content
•
Ensure stem and options are on same page
© 2014 Rollant Concepts, Inc.
#
Assembling a Test
•
Print on one side of the page – if two sided use 2 staples like a book
•
Provide margins for making notes – these can be reviewed for areas that students may have ‘fuzzy’ thinking
© 2014 Rollant Concepts, Inc.
#
Are you tense?
© 2014 Rollant Concepts, Inc.
#
© 2014 Rollant Concepts, Inc.
10
© 2014 Rollant Concepts, Inc.
11
© 2014 Rollant Concepts, Inc.
12
Test Construction
• Curriculum = 100, 200, 300, 400 • Within a Course – Formative Evaluation = – Summative Evaluation = – Exam Matrix for curriculum/course – Exam Blueprint • Revisions – Verbs for Objectives – Types of questions = structure/level • Decision Points © 2014 Rollant Concepts, Inc.
13
Test Analysis
Thought for the day… “There are three kinds of lies: lies, damned lies and statistics.”
» Mark Twain
Statistical findings in isolation are meaningless . Informed interpretation is the goal without distortion.
© 2014 Rollant Concepts, Inc.
14
Where to Analyze?
• #1 Exam – overall [KR-20] • #2 Items for Correct Responses • #3 Items for discrimination • #4 “Option A” as incorrect select • #5 OPTIONAL – Option discrimination © 2014 Rollant Concepts, Inc.
15
Summary – Steps to Analysis
1. Exam – desired KR-20 of 0.65 to 0.85 2. & 3. Items ** – mark & review ? that have
P-value of < 0.30 or > 0.80
Point Biserial Index [PBI]
• •
If negative check first [may be keyed incorrectly] Then if time review items with <0.09
**Make decisions to reject ? or accept more than one answer 4. Option Choices **
– mark & review PBI if Correct option [-] negative
Incorrect option [+] positive
© 2014 Rollant Concepts, Inc.
16
What to Analyze – the EXAM?
STEP #1 Look the Exam – overall
– Reliability coefficient [KR – 20] – desired -
KR-20 of 0.65 to 0.85
– OPTIONAL: Score distribution –
histogram
• Review raw scores and percentages • Positive or negative sided =
red flag
© 2014 Rollant Concepts, Inc.
17
© 2014 Rollant Concepts, Inc.
18
What to Analyze - the ITEM?
STEP #2 Look at the Items – Are the questions acceptable?
– Item difficulty level [
P-value responses to an item] = # of correct
– P-value of < 0.30 or > 0.80
STEP #3
– Discrimination •
Item discrimination ratio [IDR] = % of upper 27% minus the lower 27% who answered item correctly
• Point Biserial Index [PBI
] = considers the total group variance
© 2014 Rollant Concepts, Inc.
19
Step #2
© 2014 Rollant Concepts, Inc.
Step #3
20
Step #3
© 2014 Rollant Concepts, Inc.
21
What to Analyze – in Options?
STEP #4 Options – Which were selected by test-taker?
Distractor analysis
– Response frequencies & patterns
option = # number of students who selected each
– Who selects option “A” as a pattern for
incorrect items?
© 2014 Rollant Concepts, Inc.
22
Step #4
© 2014 Rollant Concepts, Inc.
23
What to Analyze – in Options?
STEP #5 Options – Which were selected by test-taker?
Distractor analysis - Point biserial for options the option = indicated the average performance of those selecting
Positive [+] correlates w/ correct option Negative [-] correlates w/ other options
© 2014 Rollant Concepts, Inc.
24
© 2014 Rollant Concepts, Inc.
25
Summary – 3 Steps to Analysis
1. Exam – desired KR-20 of 0.65 to 0.85 2. Items ** – mark & review ? that have
P-value of < 0.30 or > 0.80
• •
Point Biserial Index [PBI] negative first [check for being correctly keyed] Then <0.09 if time 3. Options **
– mark & review PBI if Correct option [-] negative
Incorrect option [+] positive **Make decisions to reject ? or accept more than one answer
© 2014 Rollant Concepts, Inc.
26