Transcript Strengthening Multiple Measures Assessment_SSSC13
Harnessing the Power of Alternative Assessment to Advance the Completion Agenda
Eloy Ortiz Oakley Superintendent-President, Long Beach City College Terrence Willett Director of Planning and Research, Cabrillo College/ RP Group Researcher Patrick Perry Vice Chancellor, CCCCO Strengthening Student Success Conference Ocotber 9, 2013
1
• • • •
Transition to College: Assessment and Placement
CCCs are open enrollment • Requires assessing and planning for students’ educational needs.
Most colleges rely on standardized assessment Most students placed into basic skills
Placement into basic skills is significant barrier to completion
2
LBCC’s Research
• • • Examined five cohorts of more than 7,000 LBUSD grads who attend LBCC directly after high school Determined strongest predictors of success in college courses Improved alignment of assessment and placement with evidence based predictors of success 3
Key Findings
1. Standardized tests predict standardized tests very well but… 2. Classroom performance predicts
classroom performance
3. More information tells us more about our students’ likelihood to succeed
Findings presented real opportunities to significantly improve student placement and achievement, and their college experience
4
Reimagining the Transition to College: Fall 2012 Promise Pathways
• Predictive placement model using multi method, evidence-based assessment • Prescriptive, full-time course load via first semester success plan – – Emphasis on Foundational Skills English, Reading, Math 5
70%
Alternative placement: Transfer-level Placement Rates 60%
60% 50% 40% 30% 20% 10% 11% 13% 7% 9%
31%
0% Transfer Level English F11 First time students F11 LBUSD Transfer Level Math F12 Promise Pathways 6
F2012 Promise Pathways 1-year vs. F2006 6-year rates of achievement
45%
41,4%
40% 35%
35,9%
31,2% 30% 27,3% 25% 20% 15% 10% 5% 0% 5,1%
15,0%
9,9% 11,9% 12,9% Successfully Completed Transfer Math F2011 LBUSD (N=1660) Successfully Completed Transfer English F2012 Promise Pathways (N=976) Behavioral Intent to Transfer F2006 LBUSD 6 Year Rate 7
Success rates in transfer-level courses Fall 2012
70% 64% 62% 60% 55% 51% 50% 40% 30% 20% 10% 0% Transfer Level English Transfer Level Math Non-Pathways (n=2095 English, 1803 Math) Promise Pathways (n=516 English, 156 Math) Neither of these differences approach significance, p >.30
8
Equity Impact of Promise Pathways: F2011 Baseline Equity Gaps
70% 60% 50% 40% 30% 20% 10% 0% 36% 29% 31% 1% 11% 4% 10% 11% 7% 11% 22% 5% 21% 22% 11% 18% Successfully Completed Transfer Math F2011 Black Successfully Completed Transfer English F2011 Asian Behavioral Intent to Transfer F2011 Hispanic Full-time, full year F2011 White
Equity Impact: F2012 Students of Color Compared to F2011 White Baseline
70% 60% 50% 40% 30% 20% 10% 0% 6% 15% 13% 10% 28% 44% 40% 22% 28% 39% 33% 22% 53% 65% 62% 31% Successfully Completed Transfer Math F2012 Black Successfully Completed Transfer English F2012 Asian Behavioral Intent to Transfer F2012 Hispanic Full-time, full year **F2011 White
Equity gaps remain but significant gains for all F2012 Students
70% 60% 50% 40% 30% 20% 10% 0% 6% 15% 13% 29% 28% 44% 40% 57% 28% 39% 33% 52% 53% 65% 62% 60% Successfully Completed Transfer Math F2012 Black Successfully Completed Transfer English F2012 Asian Behavioral Intent to Transfer F2012 Hispanic Full-time, full year **F2011 White
70%
Cohort 1 and 2 Promise Pathways Demographics
66% 60% 53% 50% 40% 30% 20% 10% 15% 10% 16% 12% 15% 10% 0% African American Asian/PI/Filipino F2012 (N=976) Hispanic White F2013 (N=1321) 2% 2% Unknown
A BRIEF OVERVIEW
The Student Transcript Enhanced Placement Project (STEPS)
Harnessing the Power of Alternative Assessment to Advance the Completion Agenda 13
What Are We Seeking to Do?
• • • • Examine the value of using transcripts as part of the assessment process Create predictive models using intersegmental data to study students who had already taken community college courses Use models to analyze how well transcript data predicts the first English & math courses students take and how well they do in them Recruit colleges to do local analyses to generate campus-specific insights and trigger CCC/K-12 conversations 14 Harnessing the Power of Alternative Assessment to Advance the Completion Agenda
In English
tests predict tests,
grades and courses matter but vary by college
15
Predicting College English Level
11 10 9 8 3 2 1 0 7 6 5 4 CST's A-G Courses HS Course Level HS Course Grade HS GPA*
Predictor Variable Category
Cox & Snell pseudo R-square ~ 0.35
Harnessing the Power of Alternative Assessment to Advance the Completion Agenda Weakest Intermediate Strongest 16
In math
tests predict tests,
but high school level is also important
17
Predicting College Math Level
11 10 9 8 3 2 1 0 7 6 5 4 CST's A-G Courses HS Course Level HS Course Grade HS GPA*
Predictor Variable Category
Cox & Snell pseudo R-square ~ 0.50
Harnessing the Power of Alternative Assessment to Advance the Completion Agenda Weakest Intermediate Strongest 18
In English,
grades predict grades
19
11 10 9 8 3 2 1 0 7 6 5 4
Predicting College English Success
Weakest Intermediate Strongest CST's A-G Courses HS Course Level HS Course Grade HS GPA*
Predictor Variable Category
College Course Level Cox & Snell pseudo R-square ~ 0.20
Harnessing the Power of Alternative Assessment to Advance the Completion Agenda 20
In math,
success predictors vary by college
21
Predicting College Math Success
11 10 9 8 3 2 1 0 7 6 5 4 CST's A-G Courses HS Course Level HS Course Grade HS GPA*
Predictor Variable Category
College Course Level Cox & Snell pseudo R-square ~ 0.20
Harnessing the Power of Alternative Assessment to Advance the Completion Agenda Weakest Intermediate Strongest 22
• • •
What Does This Mean?
There is a disconnect between test scores and performance in the first course taken at many institutions Using transcript elements would help to refine assessment The relative weight of variables might be influenced by local factors, due to considerations such as articulation and curriculum 23 Harnessing the Power of Alternative Assessment to Advance the Completion Agenda
• • • •
What Happens Next?
Updating the statewide analysis Encouraging more colleges to replicate the study so they understand the value of transcript data for their own students Starting conversations about how the study relates to other aspects of developmental education reform —such as alignment, course offerings, and curriculum Helping to develop a tool that enables colleges to access high school data to inform multiple measures assessment 24 Harnessing the Power of Alternative Assessment to Advance the Completion Agenda
Chancellor’s Office Multiple Measure Warehouse Project
Patrick Perry Vice Chancellor of Technology, Research & Information Systems
Common Assessment Project
• AB743 passed in 2011 (AB743, Block): – BOG shall establish a common assessment system to be used as one of multiple measures for placement and advisement – Ongoing funding provided in 2013-14 budget for project – RFA to move $$ to district currently underway
Common Assessment Project
• Project intends to: – Identify a common, systemwide assessment for math, English, ESL – Provide the assessment at no cost to districts for unlimited usage – Provide for score portability statewide – Create a multiple measures data warehouse to enhance placement activities
Multiple Measures Warehouse
• Will include: – Outcomes of the common assessments – CCC statewide MIS Enrollment data – EAP scores – CAHSEE scores – All HS math and English enrollment records, GPA’s • (likely through 11 th grade)
Multiple Measures Warehouse
• • Application will be written allowing a student “placement profile” to be viewed by student services personnel “big data” predictive analytics applied to multiple measures variables to assist in placement effectiveness
Contact Information
• • •
Eloy Ortiz Oakley
Terrence Willett
Patrick Perry
• [email protected] 30
Additional Resources
• • • • • More information about Long Beach City College's research – http://bit.ly/PathwaysResearch More information about the RP Group’s Student Transcript Enhanced Placement (STEPS) Project – http://bit.ly/RPSTEPS Step by Step process for replicated the research and placement: – http://bit.ly/RPSTEPS2 Jobs for the Future Report: Where to Begin?: The Evolving Role of Placement Exams for Students Starting College – http://bit.ly/JFFReport CCRC research on Assessment, Placement, and Progression in Developmental Education – http://bit.ly/CCRCAssess 31