Transcript Evaluating a NonRandomized Trial: A Case Study of a Pilot to Increase PreCollegiate Math Course Success Rates Andrew LaManque, PhD Supervisor, Institutional Research De.

Evaluating a NonRandomized Trial:
A Case Study of a Pilot to Increase PreCollegiate Math Course Success Rates
Andrew LaManque, PhD
Supervisor, Institutional Research
De Anza College
De Anza Math Pilot Results, 7-9-07
1
Outline
• Overview of the Pilot
• Purpose of the Evaluation
• Results from Quantitative Analysis
• Demographics of Students
• End of Class Survey
• Course Success
• Findings
De Anza Math Pilot Results, 7-9-07
2
Overview of the Pilot
• A Partnership was formed to increase the
success rates in Pre-Collegiate math courses
• Components:
• Math and Counseling faculty
• EnableMath – Course Software
• Noel Levitz – College Student Inventory
• Using technology, provide students timely
information on their progress as well as
information about their motivations
De Anza Math Pilot Results, 7-9-07
3
Overview of the Pilot
• Developmental math sequence consists of
pre, elementary, and intermediate algebra
• Offer 175 class sections per year of
developmental math (37% of all math
department courses)
• Historically, a 55% success rate in
developmental math courses
• 31 full-time instructors, 45 part-time
instructors
• Class size is 40
De Anza Math Pilot Results, 7-9-07
4
Program Evaluation
•Given that the pilot program provides an
additional learning choice for students, the
question from the Dean is:
“do the results demonstrate that pilot students
do at least as well as other students”
•Using multiple forms of evaluation (e.g. student
grades, student surveys) provides feedback to
instructors as well as administration on
implementation
De Anza Math Pilot Results, 7-9-07
5
Program Evaluation
Limitations
•Not a randomized trial – possible instructor
(although, 12 different instructors participated)
and student self selection bias
•No independent measure of success
•More research is needed to determine the
relative effect of each intervention
De Anza Math Pilot Results, 7-9-07
6
Results: Demographics of Students
How would you describe your Racial / Ethnic Origin?
50%
45%
40%
35%
30%
25%
20%
15%
10%
5%
0%
Twice the % of Latino students as
college, similar to non-pilot sections
African
American
Asian or
Pacific
Islander
Latino
White
Prefer not to
respond /
Other
De Anza College CSI Respondents Fall 2005-Spring 2007. N = 1,114.
De Anza Math Pilot Results, 7-9-07
7
Results: Demographics of Students
What is the highest level of education
completed by your father?
50%
45%
40%
Many first-generation
college students
35%
30%
25%
20%
15%
10%
5%
0%
high school or less
1-3 years of college
4-year degree or more
De Anza College CSI Respondents Fall 2005-Spring 2007. N = 1,028.
De Anza Math Pilot Results, 7-9-07
8
Results: Demographics of Students
•Similar with regard to Gender, Initial Education
Goal and Ethnicity
•although Math114 EnableMath had a higher
percentage of minority students (54%)
compared with the other sections (34%).
•EnableMath students are slightly older on
average (45% vs 37% born prior to 1987).
De Anza Math Pilot Results, 7-9-07
9
Results: End of Class Survey
Time Spent Studying Math at Home Compared to
Time Necessary to Get Desired Grade
100%
80%
72%
38% of respondents indicated that they needed to study more to
get the grade they wanted, than they actually studied
60%
45%
43%
40%
24%
12%
20%
4%
0%
0-4 hrs/wk outside class 5-9 hrs/wk outside class 10 plus hrs/wk outside
class
Time at Home Doing Math
Time Necessary to Get Desired Grade
De Anza College Winter 2006-Spring 2007 Term EnableMath Survey. N = 992 .
De Anza Math Pilot Results, 7-9-07
10
Results: End of Class Survey
EnableMath helped me learn more math
than I would have without the software
100%
80%
60%
77% Agree
or Strongly
Agree
40%
20%
0%
Strongly Agree
Agree
Disagree
Strongly Disagree
De Anza College Winter 2006-Spring 2007 Term EnableMath Survey. N= 986.
De Anza Math Pilot Results, 7-9-07
11
Results: End of Class Survey
Completing the CSI was worth the time
100%
80%
60%
61% Agree
or Strongly
Agree
40%
20%
0%
Strongly Agree
Agree
Disagree
Strongly Disagree
De Anza College Winter 2006-Spring 2007 Term EnableMath Survey. N=959.
De Anza Math Pilot Results, 7-9-07
12
Results: Course Success
De Anza College Math Pilot
Sections / Students, Fall 05 - Spr 07
Course
Group
# Sec
HC *
Pre-Algebra
Pilot
Other
14
55
428
1,755
Elem Algebra
Pilot
Other
21
88
723
3,045
Interm Algebra Pilot
Other
27
112
853
3,573
Total
62
255
2,004
8,373
Pilot
Other
Approx 1 of
5 students
* HC for the Pilot only includes students
completing the CSI.
De Anza Math Pilot Results, 7-9-07
13
Results: First Course Success
Course Success of Fall 2005-Spring 2007 EnableMath / Noel Levitz Sections
Compared to All Other Sections, De Anza College
Course Group
Pre Algebra
Pilot
Other
Elementary Algebra
Pilot
Other
Intermediate Algebra
Pilot
Other
Pass
Grades Percent
Did Not Pass
Grades Percent
Withdrew
Grades Percent
Total
Grades Percent
308
1,012
72%
58%
76
391
18%
22%
44
352
10%
20%
428
1,755
100%
100%
514
1,660
71%
55%
138
664
19%
22%
71
721
10%
24%
723
3,045
100%
100%
641
1,977
75%
55%
140
675
16%
19%
72
921
8%
26%
853
3,573
100%
100%
Includes end of term grades of students enrolled at census (3rd week).
Note: Students in the Pilot group used the EnableMath Software and completed the CSI.
De Anza Math Pilot Results, 7-9-07
14
Results: Second Course Success
De Anza College EnableMath Pilot, Fall 2005-Winter2007
Second Course Pass Rate for Students Successful in First Course
First Course
Pass
Course Group
Grades % Row
Pre to Elementary Algebra
Pilot
94
45%
Other
305
55%
Elementary to Intermediate Algebra
Pilot
168
64%
Other
545
66%
Intermediate Algebra to College
Pilot
85
50%
Other
301
56%
Did Not Pass
Grades % Row
W
Grades % Row
Total
Grades % Row
59
106
28%
19%
57
148
27%
26%
210
559
100%
100%
45
137
17%
17%
49
146
19%
18%
262
828
100%
100%
36
97
21%
18%
48
142
28%
26%
169
540
100%
100%
Note: Considers only the first attempt of both courses.
De Anza Math Pilot Results, 7-9-07
15
Results: Second Course Success
De Anza College EnableMath Pilot, Fall 2005-Winter2007
Second Course Pass Rate for Students Successful in First Course by Group
Course
First Course Second Course
Group
Group
Pre to Elementary Algebra
Pilot
Pilot
Control
Control
Pilot
Control
Elementary to Intermediate Algebra
Pilot
Pilot
Control
Control
Pilot
Control
Pass
Did Not Pass
W
Total
Grades % Row Grades % Row Grades % Row Grades % Row
60
34
52%
36%
36
23
31%
24%
20
37
17%
39%
116
94
100%
100%
47
258
64%
53%
14
92
19%
19%
12
136
16%
28%
73
486
100%
100%
97
71
80%
50%
17
28
14%
20%
7
42
6%
30%
121
141
100%
100%
95
450
74%
64%
24
113
19%
16%
9
137
7%
20%
128
700
100%
100%
De Anza Math Pilot Results, 7-9-07
16
Findings
• Students currently in the project want to
take next class in sequence
• These classes are filled before other
math classes
• Participating instructors & counselors
want to continue
• More faculty are wanting to participate
De Anza Math Pilot Results, 7-9-07
17
Findings
• No increased costs – taking existing
resources and using them differently
• Regular class period / time
• Counseling time in the class instead of
counseling center
De Anza Math Pilot Results, 7-9-07
18
Findings
• Overall, the data suggests that there is value
in offering a different approach to students,
but claims of improved learning, especially
the long term retention of material, can not be
substantiated.
• Difficult to evaluate a non-randomized trial
• Math department might focus on improving
the transition in pilot sections from PreAlgebra to Elementary Algebra
De Anza Math Pilot Results, 7-9-07
19
Questions
For a copy of this presentation
please email
Andrew LaManque at:
[email protected]
De Anza Math Pilot Results, 7-9-07
20