The Utility of High School Achievement Data for Placement in Foundational Skills Courses Promise Pathways: John Hetts 2012-2013 Mark Taylor Board of Trustees Retreat Andrew Fuenmayor July Cohort 24, 2012 Karen Rothstein Long Beach City College Daniel.

Download Report

Transcript The Utility of High School Achievement Data for Placement in Foundational Skills Courses Promise Pathways: John Hetts 2012-2013 Mark Taylor Board of Trustees Retreat Andrew Fuenmayor July Cohort 24, 2012 Karen Rothstein Long Beach City College Daniel.

The Utility of High School
Achievement Data for
Placement
in Foundational
Skills Courses
Promise
Pathways:
John Hetts
2012-2013
Mark Taylor
Board of Trustees Retreat
Andrew
Fuenmayor
July Cohort
24, 2012
Karen Rothstein
Long Beach City College
Daniel Lamoree
Mt. San Antonio College
RP Group Annual Conference, April 1, 2013
1
Five goals
1. Initiate reflection on student transition
to college, assessment, & placement.
2. Describe our research.
3. Introduce its implementation: F2012
LBCC Promise Pathways.
4. Briefly showcase tools Andrew, Dan,
and Karen have built to help you.
5. Help collectively unlock the potential
to better understand our students and
improve our colleges.
2
Transition to College:
Assessment and Placement
• CCCs are open enrollment
• Requires assessing and planning for
students’ educational needs.
• Most colleges rely almost entirely on
standardized assessment
•
•
•
•
Ease of administration
Transparency of administration
Impartiality of administration
Pre-approval
3
Consequences for student
transition to college
•
Majority of students placed into basic skills
•
•
Colleges’ first interaction with students is “You
are not ready for college”
•
•
Placement is significant barrier to completion
Courses take first few weeks to convince students
Substantial long-term systemic changes:
•
•
More, longer, slower remediation sequences
Reduction of access, especially at bottom of
sequences
4
Conventional Wisdom
Explaining Assessment Results
• It is a problem with today’s students
– Students are simply, vastly unprepared for
college
– Kids these days ….
• It is a problem with public education
– Public education is failing to prepare high
school students
– Teachers these days…
5
What If the Conventional
Wisdom is Wrong?
• Flynn Effect – substantial, long-term increase in IQ
• 18-24 year olds with HS degree: ~90%
• NAEP: Scores are at or near highs in virtually every
age category for every demographic group
• Growing body of research questions effectiveness
of standardized assessment for placement
– Little relation to college course outcomes. (e.g.,.
Belfield & Crosta, 2012; Edgescombe, 2011; Scott-Clayton, 2012;
Scott-Clayton & Rodriguez, 2012; Xu, forthcoming)
– NAGB, 2012: Highly variable cutscores, with 2-year
colleges often using HIGHER cutscores than 4-year
6
We sought local answers to
three questions:
• What predicts how students assess
and place into developmental
courses?
• What predicts how students perform
in those courses?
• How well are placement and
performance aligned?
7
Our Research
Five longitudinal cohorts tracking more than
30,000 students built with help of Cal-PASS
• Focusing on 7,000 HS grads who attend LBCC
directly after high school
• Examined predictive utility of wide range of high
school achievement data
•
•
•
•
most notably 11th grade California Standards Test (CST)
scores and high school grades
Assessment and placement into developmental
skills courses
Performance in those classes
8
Alignment in English
Predicting Placement
1.4
1.34x
Predicting Performance
1.0
1.4
1.2
1.0
1.2
0.8
1.0
0.8
0.6
0.8
0.6
0.6
0.4
0.4
.30**
0.2
0.0
.00
CST ELA (z) Eng Grade (12) GPA (other)
0.4
0.2
0.2
1.34x
.88x
.37***
.30**
.17*
.00
0.0
CST ELA (z) Eng Grade (12) GPA (other)
* p <.05 **, p <.01, *** p<.001, x = p< 1 x
10-10
9
Alignment in Math
Predicting Placement
Predicting Performance
1.0
0.8
1.0
.75x
0.8
0.6
0.6
0.4
0.4
0.2
.20
0.2
.73x
.20*
.25**
.21*
.07
0.0
.00
CST Math Math GPA GPA (12)
(z)
0.0
GPA (11)
CST Math Math GPA GPA (12)
(z)
* p <.05 **, p <.01, *** p<.001, x = p< 1 x
10-10
GPA (11)
10
The Key Takeaways
• We want to use assessments that predict
how students will perform at our college
– Standardized tests predict standardized tests
– Classroom performance predicts classroom
performance
– More information tells us more about our
students than less information
• Significant opportunities exist to improve
placement, student achievement, and
students’ college experience
11
Reimagining the Transition to College:
Fall 2012 Promise Pathways
• Predictive placement model in English & Math
using multi-method, evidence-based assessment
– English: A or B in 12th Grade English (as proxy)
– Math: Predicted rate of success using all
variables ≥ average success rate in course
• Prescriptive, full-time course load via prepopulated first-semester success plan
– Emphasis on English, Reading, Math
• Required group Counseling workshop
• Student success course
• Additional Experimental pilots (Contextualized Reading,
academic achievement coaches)
12
Alternative placement:
Transfer-level Placement Rates
70%
60%
60%
50%
40%
31%
30%
20%
11%
13%
7%
10%
9%
0%
Transfer Level English
2011 First time students
Transfer Level Math
2011 LBUSD
Promise Pathways
13
First-term early educational milestone
achievement
70%
62.9%
60%
52.9%
50%
40%
37.1%
32.8%
30%
20%
10%
16.0%
5.2%
8.2%
2.8%
5.5%
3.8%
8.2%
2.0%
0%
Attempted Successfully Attempted Successfully Behavioral Behavioral
Transfer Completed Transfer Completed Intent to
Intent to
Math
Transfer
English
Transfer
Transfer
Complete
Math
English
F2011 LBUSD (N=1660)
F2012 Promise Pathways (N=976)
14
Success rates in transfer-level courses
Fall 2012
70%
64%
62%
60%
55%
51%
50%
40%
30%
20%
10%
0%
Transfer Level English
Non-Pathways
Transfer Level Math
Promise Pathways
Neither of these differences approach significance, p >.30
15
How can it affect time to collegelevel work?
Average number of semesters to
take English 1 by F2012 NonPathways Students by placement
7
6.5
6
3000
2541
2500
5.2
5
2000
4
3
Estimate of number of semesters
saved among F2012 Pathways English
1 Students by placement
1500
2.7
1294
983
1000
2
1
500
0
0
Transfer-level
One level
below
Three levels
below
265
Transfer-Level One Level
(N=98)
Below
(N=189)
Three Levels Total (N=486)
Below
(N=199)
1270 years of student time saved, or approximately 2.6 years per student
16
Can it be done?
• Can your college do this research?
• Could this model apply at your
college too?
• Can your college afford to
implement something like this?
17
If you’re interested in doing this research too:
When you start this research:
Step 1: Acquiring Cal-PASS Data
• Log in at mycalpass.org and click on the data menu.
Step 1
18
Raw Cal-PASS Data
Step 1
19
Downloading and Running
the STEPS 2.0 Module
Step 2
20
How is STEPS 2.0 different?
• Interface
– Redesigned the initial STEPS Database using
VBA, SQL, and Shell scripts to fully automate
data management
– Forms to facilitate data validation
• High School Course Id (CBEDS code)
• Prior to College Level (MIS: CB21)
• To previous Spring to Fall direct
matriculants, dataset adds students:
– enrolled in summer term immediately after
graduation
– enrolled for first time in subsequent terms
– not enrolling in MATH or ENGL in their first term
at your institution
*Note: You may restrict the dataset to fully mirror the original methodology!
Step 2
21
Who is in the dataset?
1
• Students from your local feeder high schools
2
• Their most recent high school academic year (MATH or ENGL)
3
• Their most recent high school term (MATH or ENGL)
4
• Their highest high school course within subject (CBEDS)
5
• Their highest grade
6
• Their most recent term at your institution
7
• Their lowest level (CB21) course within ToP code (MATH or ENGL)
8
• Their highest grade
Step 2
22
Final Analysis: Placement
Step 3
23
Predicting Placements
Step 3
24
Final Analysis continued:
Performance
Step 3
25
Predicting Success in English
Step 3
26
Using the Model to Develop
Alternative Placements
• To compute this measure we capture all the coefficients from our
Predicting Success model.
Step 3
27
Using the Model to Develop
Alternative Placements (continued)
.6
.4
.6
.4
Step 3
28
So, can it be done?
• Can your college do this research?
– YES! (Still skeptical? Come see Dan demo his even
newer and more improved tool at 1:45 tomorrow in
Mountain Vista)
• Could this model apply at your college too?
– YES! (Not sure? Join us for a panel discussing similar
research and two statewide pilots, 10:15 AM Campus
Vista)
• Can your college afford to implement something like
this?
– Can your college afford not to?
• But wait, there’s more…
– Tools to help understand effects of other interventions
– Broad new understandings of our students and
education
29
Improved tools to understand
student success interventions
(Come find out more at 9AM, Valley Vista tomorrow)
Success Rate in Term Following Target Course
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
71.8%
65.7%
61.5%
40.3%
46.4%
Successful in Unsuccessful in Did not enroll Successful in Unsuccessful in
target course target course
in target
comparison
comparison
course
course
course
30
Powerful start but self-selection questions
remain. However, if supplemented by data
generated by these tools…
Outcome: Success in courses in the following term
B
Passed target course
(not controlling for high school)
Passed target course
(controlling for high school GPA)
p
0.124
0.039 (significant)
0.037
0.550 (not significant)
31
Possible effects on how we
understand college readiness
College Readiness – All 2011-2012 LBUSD Seniors
60%
50%
40%
Plus LBUSD Alternative
Placement for all
30%
Plus LBCC (Accuplacer &
Alternative Placement)
30%
27%
20%
10%
EAP - LBUSD
8%
18%
6%
6%
0%
English
Math
32
What did LBCC gain through
prescriptive, evidence based
approach to transition to college
– Dramatic increases in students attaining early
educational outcomes, & shorter times to do so
– New discussion of research and instructional
pedagogy, kick-starting experimentation and
innovation
– Strong challenge to conventional wisdom and
perceptions of our students by administration,
staff, faculty, and students themselves
– Most importantly, concrete achievable steps
that any college can take to dramatically
improve all of our students’ futures
33
Can you get in touch with us?
• General questions
– Mark Taylor, [email protected] or (562)938-4206
• Research questions
– John Hetts, [email protected] or (562) 938-4052
• STEPS Participation
– Terrence Willett , [email protected] or (510)
527-8500, ext. 109
• STEPS syntax/output questions
– Andrew Fuenmayor, [email protected], or
(562) 938-4133
• STEPS 2.0/Illuminous questions?
– Dan Lamoree, [email protected] or
(909)274-4708
34
Resources
• More information about our research
– http://bit.ly/PathwaysResearch
• Achieving the Dream/Jobs for the Future
summary of alternative assessment
– http://bit.ly/AlternativeAssessment
• CCRC research on Assessment, Placement, and
Progression in Developmental Education
– http://bit.ly/CCRCAssess
• More information about the RP Group’s Student
Transcript-Enhanced Placement (STEPS) Project
– http://bit.ly/RPSTEPS
• Step by Step process for replication
– http://bit.ly/RPSTEPS2
35
36
Completion Rates: English
80%
68%
70%
67%
60%
50%
52%
47%
40%
35%
30%
20%
9%
10%
0%
Transfer-Level English
One Level Below
Two Levels Below
F2008 First Time Student Cohort 4-year Completion Rate
F2012 Promise Pathways First-term completion rates of Alternative Placement Students
Completion Rates: Math
80%
70%
63%
68%
57%
60%
50%
41%
40%
27%
30%
20%
13%
10%
0%
Transfer-Level Math
One Level Below
Two Levels Below
F2008 First Time Student Cohort 4-year completion rate
F2012 Promise Pathways Alternative Assessment (First-term completion rate)
38