Transcript Assessment test validity English 2-12-12

English Assessment Validity
February 2012
Michael Orkin, Jo Ann Phillips, Hui Zhang, Sheryl Queen
Office of Institutional Research
Peralta Community College District
1
English Assessment Test
• New Peralta students take assessment tests
for placement in English classes.
• Course placement from test scores is
recommended but not mandatory.
• A student can take courses without
placement tests if prerequisites are met.
2
ACT Compass Test
• Peralta currently uses the ACT Compass test,
CENG, for English placement.
• CENG tests English proficiency with separate
writing and reading tests.
• This will soon change...
3
Statewide Online Assessment
• From Student Success Task Force, January,
2012:
“… all students will have access to common online
assessment tools for English, mathematics and ESL
and to pre-testing programs that help improve
assessment outcomes. They also will be able to take
the results of their tests to any community college in
the state.”
4
For now: ACT Compass Test
Placement in English Courses
A decision rule based on the two test scores
selects an English course:
• transfer level (English 1A)
• degree applicable (English 201A)
• Pre-collegiate (Basic Skills).
5
Placement in English Course
• If the writing score is less than 22, the
student is not assessed into a course and
must see a counselor.
• 699 students in the sample data had low
writing scores and were excluded.
6
Data
• Students taking CENG-recommended English
courses in Fall 2008, Spring 2009, Fall 2009,
Spring 2010.
• Took test immediately preceding semester
when they took English course.
• Never took test prior to semester when they
took English course.
7
Validity
Validity measured in three ways:
• Relationship between assessment scores and
grades.
• Comparison of assessed and non-assessed.
• Comparison of students who took the “right”
course (recommended by CENG) and
“wrong” course (not recommended by
CENG).
8
Assessed and Took Right Course
Courses
Eng 1A
Count
%
1571 57%
Eng 201A
593
21%
Basic Skills*
598
22%
TOTAL
2762 100%
* English 252AB, 267AB, 268AB, 269AB
9
Assessed - Grade %
A
Eng 1A
B
C
D
28% 23% 12% 5%
F
W
8% 25%
Eng 201A 21% 19% 15%
9%
Basic Skills 23% 22% 16%
4% 10% 24%
ALL
25% 22% 13% 6%
9% 27%
9% 25%
10
Placement and Grades
• 57% were placed in English 1A.
• 47% overall got A’s or B’s.
• 25% overall withdrew from class.
11
Grades vs. Test Scores
Are assessment test scores positively correlated
with grades?
We use ANOVA (Analysis of Variance).
12
Reading Test Scores by Grade
Courses
A - Test ave B - Test ave C - Test ave D - Test ave F/W- Test ave
ANOVA
Sig.
Eng 1A
91.61
87.33
86.98
87.01
87.88
0.000
Eng 201A
84.17
81.35
80.36
83.75
81.12
0.000
Basic Skills 61.84
58.92
57.99
60.08
57.29
0.036
13
All Test Scores by Grade
Courses
ANOVA
Sig. Reading
ANOVA
ANOVA
Sig. Writing Sig. Combined
Eng 1A
0.000
0.000
0.000
Eng 201A
0.000
0.727
0.727
Basic Skills
0.036
0.170
0.481
14
Grades vs. Test Scores Analysis
• Reading test scores most sensitive to grades.
• Students who get A’s have higher average
test scores than those who have lower
grades.
• Writing test and Combined Tests only
significant for English 1A.
15
Predicted Chance of Success
As test scores get higher, does predicted
chance of success increase?
We use logistic regression.
16
Reading Test Predicted Chance of Success
17
Reading Test Predicted Chance of Success
18
Reading Test Predicted Chance of Success
19
Predicted Chance of Success
As test scores get higher, predicted chance of
success increases but …
• Only for reading test scores.
• Different courses are selected across range of
scores so much is filled in by software.
• Results aren’t statistically significant.
20
Assessment vs. No Assessment
Do assessed students do better in their
assigned courses than students who don’t
take the assessment test?
We use ANOVA and Chi Square tests.
21
Counts
Courses
Assessed
Not Assessed
Eng 1A
1571
1838
Eng 201A
593
621
Basic Skills*
598
727
TOTAL
2762
3186
* English 252AB, 267AB, 268AB, 269AB
22
Not Assessed - Grade %
(Similar to Assessed)
A
B
C
D
F
W
24%
Eng 1A
25% 25% 15%
5%
6%
Eng 201A
18% 18% 19%
9%
11% 25%
Basic Skills 21% 18% 13%
6%
10% 31%
23% 22% 16%
6%
8%
ALL
25%
23
Assessed vs. Not Assessed GPA
Courses
Assessed Not Assessed ANOVA
Course GPA Course GPA
Sig.
Eng 1A
2.76
2.74
0.730
Eng 201A
2.47
2.32
0.099
Basic Skills
2.58
2.51
0.426
TOTAL
2.61
2.66
0.212
24
Assessed vs. Not Assessed GPA
No significant results:
• Students who are assessed have similar class GPA
as those who are not assessed.
25
Assessed vs. Not Assessed Success
Courses
Assessed % Not Assessed % Chi-Square
Success
Success
Sig.
Eng 1A
62%
65%
0.112
Eng 201A
55%
56%
0.795
Basic Skills
61%
53%
0.004
TOTAL
60%
60%
0.984
26
Success - Assessed vs. Not Assessed
Significant only at basic skills level:
• Students who are assessed have greater success
than those who are not assessed.
27
Assessed – “Right” vs. “Wrong” Course
Do assessed students who take the right
course do better than assessed students who
take the wrong course?
We use Chi Square and ANOVA.
28
Counts - Right vs. Wrong Course
Courses
Right Course Wrong Course % Right
Count
Count
Course
Eng 1A
1571
378
81%
Eng 201A
593
479
55%
Basic Skills
598
127
82%
TOTAL
2762
984
74%
29
Success - Right vs. Wrong
Courses
Right Course
Success %
Wrong Course Chi Square
Success %
Sig.
Eng 1A
62%
53%
0.001
Eng 201A
55%
52%
0.364
Basic Skills
61%
61%
0.932
30
Success - Right vs. Wrong
Significant only for English 1A:
• Assessed students who take right course have
greater success than those who take wrong
course.
31
GPA - Right vs. Wrong
Courses
Right
Wrong
ANOVA
Course GPA Course GPA
Sig.
Eng 1A
2.76
2.39
0.000
Eng 201A
2.47
2.34
0.178
Basic Skills
2.58
2.58
0.986
32
GPA - Right vs. Wrong
Significant only for English 1A:
• Assessed students who take right course have
higher GPA than those who take wrong course.
33
Conclusions
• Validity confirmed.
• Useful for counselors.
• Assessed results aren’t much different from
non-assessed, pre-requisite method.
• Most students take English 1A.
• Students assessed into English 1A do
significantly better than “wrong” students.
34