Transcript Using a Regression Discontinuity Design to

Using a Regression Discontinuity Design
to Estimate the Impact of Placement
Decisions in the
Los Angeles Community College District
Tatiana Melguizo & Bo Kim (USC)
Hans Bos (American Institutes for Research)
George Prather (LACCD, Retired)
Presented at the Research and Planning Group conference, April 2, 2013
This research is funded by a grant from the U.S. Department of
Education’s Institute of Education Sciences (IES).
The Problem
•
•
•
Community college students have widely
varying initial skill levels and the majority arrive
with substantial remedial needs in math
Colleges have to offer classes to meet the needs
of their diverse students and have to keep
heterogeneity in the classrooms manageable
Placing students incorrectly can reduce the
likelihood that students succeed
Math Placement in the Los Angeles
Community College District
About sixty percent of
the entering students
need at least one
remedial class before
taking a math class that
counts towards a degree
Some Controversy about Impact of
Remedial Education
•
•
•
Some research finds that remedial education
provides the preparation necessary for students
to succeed in college (Boylan, Bliss, & Bonham,
1994; 1997; Lazarik, 1997)
Critics contend that remedial education may
hold students back and be ineffective (Calcagno
& Long, 2008; Martorell & McFarland, 2011)
We argue that it depends on how students are
placed and how cut points between levels are
set
Focus on Cut Points
•
•
•
•
Math faculty set the cut points between the
different levels based on who applies and how
their course offerings are distributed
If the cut points are too high, too many students
languish in remedial courses
If the cut points are too low, too many students
fail higher-level courses and present a challenge
to the instructors
Getting the cut points just right is important
Changing Cut Scores
Different Pathways to Success
(Arithmetic vs. Pre Algebra)
Placed in
Enroll in
Pre-Algebra
Pre-Algebra
No
enrollment
Test
Placed in
Arithmetic
Enroll in
Arithmetic
Success
Failure
Success
Failure
Next
course
Success Above Cut Point
Placed in
Enroll in
Pre-Algebra
Pre-Algebra
Success
Failure
No
enrollment
Test
Placed in
Arithmetic
Enroll in
Arithmetic
Success
Failure
Direct but academically challenging
Next
course
Success Below Cut Point
Placed in
Enroll in
Pre-Algebra
Pre-Algebra
Success
Failure
No
enrollment
Test
Placed in
Arithmetic
Enroll in
Arithmetic
Success
Failure
A less academically challenging but more
time-consuming trajectory
Next
course
The Impact Question
•
•
For a student at the margin, does the longer road
to success produce better results?
Is greater likelihood of success worth the extra
time and effort?
The Policy Question
•
Are cut points between math placement options
set correctly?
Using Regression Discontinuity to
Evaluate Placements
•
•
•
Regression discontinuity analysis is the
strongest non-experimental method to estimate
causal effects
It depends on a continuous forcing variable and
an exogenously established cut point
Those two conditions are present in this
situation
Outcome
Impact
Arithmetic
Pre-Algebra
Placement Test Score
Analytical Details
•
•
Not everyone follows the placement test’s
recommendation
Some students enroll below or above their placement
level
•
•
•
Compliance was high
Impacts are estimated using discrete time survival
analysis implemented within a regression discontinuity
framework
The density of the forcing variable around the cut point
was normal (confirmed with McCrary test)
The Impact of Placement Decisions
Needs to be Observed Over Time
The Impact of Placement Decisions
Needs to be Observed by College
The Impact of Placement Decisions
Needs to be Observed by Level
Students are not Being Placed Effectively
in the Higher Levels (Elementary versus
Intermediate Algebra)
Students Placed in Lowest Levels
Accumulate More Degree Applicable
Credits
Placement in Less Effective in Terms of
Transfer Degree Credit Accumulation
Conclusions
•
•
•
•
Cut points between elementary algebra and
intermediary algebra may be set too high as
marginal students do worse in elementary
algebra
At lower levels the cut points appear to be in the
right place
For students placed in lower levels of math
sequence, there is no penalty in terms of
accumulating 30 degree applicable credits
For students placed in higher levels there is a
penalty in terms of completing 30 transfer
credits
THANK YOU!
Questions
Tatiana Melguizo
[email protected]
http://www.usc.edu/dept/education/rossier_faculty/tmelguizo/