The ABC’s of Pattern Scoring

Dr. Cornelia Orr

Vocabulary

• Measurement – Psycho metrics is a type of measurement • Classical test theory • Item Response Theory – IRT (AKA logistic trait theory) • 1, 2, & 3-parameter IRT models • Pattern Scoring Slide 2

General & Specialized

Measurement • Assign numbers to objects or events • Ex. – hurricanes, earthquakes, time, stock market, height, weight Psychometrics • Assigning numbers to psychological characteristics • Ex. – achievement personality, IQ, opinion, interests Slide 3

Different Theories of Psychometrics

Classical Test Theory • Item discrimination values • Item difficulty values (p-values) • Guessing (penalty) Number correct scoring Item Response Theory a) Item discrimination values b) Item difficulty values c) Guessing (pseudo guessing) values Pattern scoring Similar constructs – Different derivations Slide 4

Different Methods of Scoring

• • Number-Correct Scoring • Simple Mathematics • – Raw scores (# of points) Mean, SD, SEM, % correct Number right scale – Score conversions Scale scores, percentile ranks, etc. Pattern Scoring • Complex Mathematics • Maximum likelihood estimates – Item statistics, student’s answer pattern, SEM • Theta scale (mean=0, standard dev=1) • Score conversions – Scale scores, percentile ranks, etc.

Slide 5

Comparison: Number Correct and Pattern Scoring

Similarities • The relationship of derived scores is the same, e.g., – High correlation, (0.95) of number right scores and scale scores – Scale score has the same percentile rank for both methods Differences • Methods of deriving scores • The number of scale scores possible – Number right = limited to the number of items – IRT = unlimited or is limited by the scale (ex. 100-500) Slide 6

Choosing the Scoring Method

• Which model?

• Simple vs. Complex?

• Best estimates? • Advantages/Disadvantages?

Ex. – Why do the same number correct get different scale scores?

Ex. – Flat screen TV – how do they do that?

Slide 7

Advantages of IRT and Pattern Scoring

• Better estimates of an examinee’s ability – the score that is most likely, given the student’s responses to the questions on the test (maximum likelihood scoring) • More information about students and items are used • More reliability than number right scoring • Less measurement error (SEM) Slide 8

Disadvantages of IRT and Pattern Scoring

• Technical - Complex Mathematics – – Difficult to understand – Difficult to explain • Not common – Not like my experience. • Perceived as “Hocus Pocus” Slide 9

Item Characteristic Curve (ICC)

Pseudo-Guess. =0.13

-4 Discrimination=1 Difficulty =0.5

-2 0 Achievement Index (Theta) 2 4 Slide 10

Examples

Effect Of Item Difficulty

No Type a b c 1 1 MC 0.0150 250.000 0.1 2 1 MC 0.0150 275.000 0.1 3 1 MC 0.0150 300.000 0.1 4 1 MC 0.0150 325.000 0.1 5 1 MC 0.0150 350.000 0.1 Response Patterns (1=correct) Pattern SEM SS 12345 11100 01110 10101 10011 43 43 43 43 300 305 305 310 Answering more difficult items (b-parameter) can result in higher scores.

Slide 11

Examples

5 Items (Effects of Item No Type a b c 1 MC 0.0050 300.000 0.2 2 MC 0.0100 300.000 0.2 3 MC 0.0150 300.000 0.2 4 MC 0.0200 300.000 0.2 5 MC 0.0250 300.000 0.2

Response patterns (1=correct) Pattern SEM SS 12345 11001 11100 94 61 01110 46 00111 39 280 270 300 305 Answering more discriminating items (a-parameter) can result in higher scores. Slide 12