Compass Math Test Validity

March 2011

Mike Orkin, Jo Ann Phillips, Hui Zhang Office of Institutional Research Peralta Community College District 1

Compass Math Test

• Peralta students take assessment tests for placement into Math, ESL, and English courses. • Course placement based on test scores is recommended not mandatory. • A student may take courses without taking placement tests.

2

Compass Math Test

• Peralta uses the ACT Compass test for math placement.

• When students take Compass, they advance through “subtests” until they reach a “placement domain.” • The higher the placement domain, the higher the difficulty level of the recommended courses.

3

Data

• Students taking Compass-designated math courses in F08, S09, F09, S10. • Took Compass test immediately preceding semester when they took math course, or • Never took Compass test prior to semester when they took math course.

4

Data Totals

• 12,657 student enrollments. • 4,915 (39%) took Compass test.

• 7,742 (61%) did not.

5

Data by Placement Domain Courses

Placement Domain Courses Compass

MPRAL 3,749 MALGE 755 MTRIG 179 MCOAL 232

No Compass

2,644 5,268 643 3,590

Total

6,393 6,023 822 3,822

% Take Compass

59% 13% 22% 6% 6

Placement Domains and Courses

• MPRAL – Prealgebra  Arithmetic – Math 250 (0-32)  Pre-Algebra – Math 253 (33-39)  Elementary Algebra – Math 201 (40-100) 7

Placement Domains (Subtests)

• MALGE – Algebra.  Elementary Algebra – Math 201 (0-34)  Intermediate Algebra – Math 203 (35-49)  Statistics – Math 13 (50-100) 8

Placement Domains (Subtests)

• MTRIG – Trigonometry.  Trigonometry – Math 50 (0-45)  Pre Calculus – Math 1 (46-100) 9

Placement Domains (Subtests)

• MCOAL – College Algebra.  Statistics – Math 13 (0-34)  Pre Calculus – Math 1 (35-49)  Calculus – Math 3A (50-100) 10

Compliance Within Placement Domain

• Most complied with cutoff score assignment within their placement domain.

11

Compliance Within Placement Domain

% Placement Domain Compliance

MPRAL COMPLY NONCOMPLY MPRAL Total MALGE COMPLY NONCOMPLY MALGE Total MTRIG COMPLY NONCOMPLY MTRIG Total MCOAL COMPLY NONCOMPLY MCOAL Total Grand Total

Total

3,264 485 3,749 682 73 755 125 54 179 169 63 232 4,915 87% 13% 90% 10% 70% 30% 73% 27% 12

Validity

• A test is “valid” if it measures what it’s supposed to measure. • We looked at this in two ways: I. Are grades and success positively related to Compass scores?

II. Does Compass do “better” than students who don’t take Compass? 13

Validity

• Are grades positively related to Compass scores?  On average, as grades increase, so do scores.

14

Validity

ANOVA - Assessment test averages by grade.

Placement A - Test ave B - Test ave Domain C - Test ave D - Test ave F/W- Test ave MPRAL

38.49

35.77

34.41

34.97

32.58

Sig.

0.000

MALGE

48.77

44.73

39.03

43.33

38.62

41.36

37.45

44.51

36.56

0.014

0.401

MTRIG ALL

39.55

MCOAL

71.87

44.12

69.6

39.35

65.23

37.25

64.43

36.54

64.77

34.53

0.037

0.000

15

Validity

Grade Counts

Placement Domain MPRAL A

767

MALGE

239

MTRIG

69

MCOAL

127

ALL B

569 139 39 43 1,202 790

C

471 113 26 22 632

D

275 47 11 7 340

F W Total

504 1,107 3,693 70 9 10 144 752 25 21 179 230 593 1,297 4,854 16

Validity

• Do assessment test scores predict a student’s chance of success (C or better) in a course?

 Logistic regression estimates the probability of success for a given assessment test score.

 For all subtests, as scores get higher, the predicted chance of success increases, indicating predictive validity.

17

Validity

18

Validity

19

Validity

20

Validity

21

Validity

22

Validity

•  Does Compass do better than students who don’t take the test?

Assume that those who don’t take Compass essentially select a class at random.

23

Validity

• For non-Compass students, just by chance, some strong students will take lowest-level courses (too easy) and some weak students will take higher level courses (too difficult). • If Compass is properly assessing students, only weaker students will take lowest level courses and stronger students will take higher level courses. 24

Validity

• Thus, students who take Compass should do worse than those who don’t at lowest subtest and better than those who don’t at higher levels.

• This is exactly what we observe in the data.

25

Validity

Chi Square - Success – Compass Test v. No Test

Placement Domain Courses Take Test % Success No Test % Success Sig.

MPRAL

48.2% 50.9% 0.020

MALGE MTRIG MCOAL

65.0% 74.9% 82.8% 57.5% 0.000

56.9% 0.000

60.8% 0.000

26

Validity

• Similar result when comparing grades of students who take Compass with those who don’t.

27

Validity

ANOVA - Test v. No Test Course GPA

Placement Domain Courses MPRAL Test GPA No Test GPA

2.3

2.6

MALGE

2.7

2.6

MTRIG MCOAL

3.0

3.3

2.8

2.8

Sig.

0.000

0.245

0.083

0.000

28

Summary

• All four methods of analysis yield statistically significant results supporting Compass validity.

29

Summary

• On average, as grades increase, so do Compass scores.

• As Compass scores increase, the predicted chance of success increases. • Students who take Compass do better than those who don’t in the higher subtests and worse in the lowest subtest, in both course success and grades. 30