Transcript PRELIMINARY. PLEASE DO NOT CITE OR TWEET WITHOUT AUTHORS’ PERMISSION Beyond Triage: A Randomized Experiment in Sustained Pre-College Advising E R I C.

PRELIMINARY. PLEASE DO NOT CITE OR TWEET WITHOUT AUTHORS’ PERMISSION
Beyond Triage: A Randomized Experiment in
Sustained Pre-College Advising
E R I C P. B E T T I N G E R
B R E N T J . E VA N S
W I T H A N T H O N Y A N TO N I O , J E S S E F O S T E R , E I L E E N H O R N G , R I E K I J I M A
U C D AV I S – 1 5 M A R C H 2 0 1 5
Motivation
Non-financial barriers related to information deter students from preparing for and enrolling in
higher education, especially among low-income and minority students (Avery & Kane, 2004)
Gauntlets
◦ Financial aid (ACSFA, 2005)
◦ Admission process (Klasik, 2012)
High school counselors are overloaded and cannot meet the demand in their schools.
Huge investment by college access programs to provide information and guide students through
the process
◦ TRIO (> $800 million)
◦ Countless small local programs
Models vary substantially
Literature
Information and guidance matter at very specific stages in the process
◦ H&R Block Study (Bettinger, Long, Oreopoulos, & Sanbonmatsu, 2012)
◦ Information and guidance related to financial aid improves aid receipt and college enrollment
◦ 2-8 percentage point effect
◦ Summer Melt Studies (Castleman, Page, & Schooley, 2014; Castleman, Arnold, & Wartman, 2012)
◦ Outreach to students in the summer between high school completion and college enrollment improves likelihood of enrolling
◦ 3-14 percentage point effect
Holistic advising
◦ Mathematica’s Upward Bound Study (Myers et al., 2004)
◦ Primarily academic supports to promote college enrollment, but also provide information
◦ RCT found no effects on college enrollment although some evidence of shift from two-year to four-year
◦ College Possible (Avery, 2013)
◦ Targeted ACT prep, information, and admission support
◦ High-touch intervention-360 hours
◦ Large effects (15 pp) on shifting from two-year to four-year college but no overall enrollment effects
◦ Dartmouth Outreach (Carrell and Sacerdote, 2013)
Need for evidence of different models
Different models of providing information and guidance
◦ Targeted versus school wide
◦ In school versus out of school
◦ Near-peer
By investigating wide range of models, we can start to tease apart mechanisms
Need well identified, causal estimates of efficacy
Randomized controlled trial evidence should be privileged where possible
Advise TX
Part of the College Advising Corps model (active in 15 states 350-400 high schools)
Goal is to help low-income, first-generation college students attend and succeed in college
Recent college graduates serve as full-time near-peer advisers to entire student body at a high
school
Full school college advising intervention-concentrate on seniors but some time spent developing
college plans for underclassmen
Encourage attendance; provide information; assist with selection of colleges, college
applications, and FAFSA completion
Program pilot in 2010-2011 in 15 schools with rapid expansion to over 120 schools in 2011-2012
school year
Experimental Design
School level randomization
Schools invited to apply
◦ > 35% FRL
◦ < 70% graduates attend college within one year (average is 45% in 2009)
◦ < 55% students undertaking a “distinguished” college-prep curriculum
Schools were ranked on these 3 criteria and a qualitative “fit” component
84 schools automatically selected among 237 who applied
Next 111 schools eligible for random assignment (experimental sample)
36 randomly selected for treatment assignment, others assigned control status (no program)
Blocked on region: individual lotteries were held within each region (23)
Data
School level data from Texas Education Agency (TEA)
Student level data from Texas Higher Education Coordinating Board (THECB)
2 years of enrollment outcome data
◦ From 2011-2012 (2012) and 2012-2013 (2013) high school graduating classes
◦ All in-state public postsecondary institutions
◦ Our results are lower bounds of effect of program if the program had any effect on out of state or
private enrollments
◦ Outcomes: 2-year and 4-year enrollment and enrollment in Fall after high school graduation
Preliminary results from student surveys conducted in treatment/control schools
Treatment-Control Balance
Panel A: School Level
Variable
All TX High Schools
Mean
Stdev.
All Experiment High
Schools
Mean
Stdev.
All Treatment High
Schools
Mean
Stdev.
Raw Difference T-C
Difference
P-value
T-C Difference with Lottery
Controls
Difference
P-value
White
0.400
0.311
0.225
0.227
0.214
0.223
-0.017
0.718
-0.015
0.665
Black
0.123
0.177
0.171
0.171
0.217
0.208
0.068
0.049
0.085
0.006
Hispanic
0.440
0.313
0.568
0.273
0.526
0.276
-0.062
0.266
-0.083
0.033
Asian
0.017
0.045
0.021
0.029
0.029
0.037
0.011
0.060
0.013
0.017
Other race
0.021
0.041
0.015
0.016
0.014
0.011
-0.001
0.783
0.000
0.948
Low-income
0.542
0.258
0.635
0.179
0.636
.0169
0.001
0.980
0.006
0.843
Grad Rate*
0.804
0.217
0.800
0.100
0.802
0.096
0.003
0.899
0.014
0.470
Total Students
728
891
1683
838
1848
956
243
0.154
194
0.210
Total Seniors
159
209
370
187
418
220
71.8
0.058
62.2
0.066
N
1785
111
36
Descriptive Statistics & Balance
Relative to all TX high schools, experimental schools
◦
◦
◦
◦
Are much larger (1683 to 728)
Have a higher percentage of low-income status (64% to 54%)
Have a higher share of minority students (77.5% to 60%)
Have same graduation rate (80%)
Treatment schools relative to control schools
◦
◦
◦
◦
Have more Black and less Hispanic students (8 percentage points difference)
Have slightly more Asian students (1 percentage point difference)
Same graduation rates and college enrollment rates
We control for race in all models to account for this imbalance
Compliance
5 of 36 treatment schools declined to participate or eventually left the program
9 of 75 control schools eventually received treatment
◦ Although we had a randomly determined waitlist, it was not consistently adhered to by program staff
◦ E.g. Austin loved the program and offered to pay for all control and treatment schools to participate
after the first year.
Treatment on the treated estimates are approximately 33% higher than Intent to treat estimates
Treatment Received
Control Received
Total
Treatment Assigned
31
5
36
Control Assigned
9
66
75
Total
40
72
111
Lottery controlled regression of treatment
received on treatment assignment
0.745
(0.072)
Main Impacts
Panel B: Separate Treatment Years
Outcomes
Enrolled in Higher
Education Fall after
HS
Treatment Year Model 1
2012
0.021 *
(0.0102)
Model 2
0.0146
(0.0092)
Model 3
0.0146
(0.0090)
2013
0.0133
(0.0105)
0.0089
(0.0094)
0.0089
(0.0092)
Enrolled in 2 Year Fall 2012
after HS
0.019
(0.0118)
0.0208 +
(0.0112)
0.0208 +
(0.0110)
2013
0.0061
(0.0122)
0.0084
(0.0114)
0.0084
(0.0112)
Enrolled in 4 Year Fall 2012
after HS
0.0053
(0.0082)
-0.0029
(0.0083)
-0.0029
(0.0083)
2013
0.0077
(0.0082)
0.0011
(0.0076)
0.0011
(0.0076)
X
X
X
77823
77823
Controls
SE Clustered at
SchoolxYear Level
N
77823
Intent to treat estimates
1.2 – 1.5 pp impacts on enrolling in fall after high school completion (significant at 10% level)
◦ Impacts are observed in first year of treatment
◦ enrollment effects on two-years become more significant if just examine first year
If we pool the data, small and insignificant effects (~ 1 pp) on overall enrollment
◦ Concentrated at 2-years
◦ Point estimates at 4-years close to 0
All second year impacts attenuated
Impacts on Subgroups
Subgroup
Outcomes
Coefficient
Female
White
Black
Hispanic
Asian
LowIncome
Enrolled in Higher
Education Fall
after HS
Treatment
Main Effect
0.0167 *
(0.0076)
0.0155 *
(0.0073)
0.0147 *
(0.0071)
-0.0028
(0.0095)
0.0124 +
(0.0064)
-0.01
(0.0090)
Interaction
Effect
-0.0100
(0.0081)
-0.0197
(0.0140)
-0.0157
(0.0139)
0.0257 +
(0.0131)
-0.0183
(0.0270)
0.0345*
(0.0120)
Treatment
Main Effect
0.0116
(0.0080)
0.0165 +
(0.0087)
0.0215 *
(0.0086)
-0.0047
(0.0103)
0.0155 +
(0.0079)
0.0041
(0.0097)
Interaction
Effect
0.0059
(0.0083)
-0.0103
(0.0127)
-0.0370 *
(0.0163)
0.0340 **
(0.0128)
-0.0264
(0.0263)
0.0166
(0.0104)
Treatment
Main Effect
0.0072
(0.0061)
0.0016
(0.0058)
-0.0045
(0.0059)
0.0015
(0.0085)
-0.001
(0.0055)
-0.0135
(0.0085)
Interaction
Effect
-0.0161 **
(0.0060)
-0.0127
(0.0147)
0.0191
(0.0133)
-0.0041
(0.0092)
0.0045
(0.0290)
0.0203 *
(0.0097)
77823
77823
77823
77823
77823
77823
Enrolled in 2 Year
Fall after HS
Enrolled in 4 Year
Fall after HS
N
Subgroup Analysis
Impact on Low-income students
◦ More likely to enroll overall, split between two-years and four-years (2-3 pp)
Impact on Hispanic students
◦ More likely to enroll overall, mostly at two-years (2-3 pp)
Impact on Black students
◦ Less likely to enroll in a two-year (3 pp)
◦ Some evidence of substitution into four-year, but four-year enrollments not significant
Impact on Female students
◦ Less likely than men to enroll in four-year and start at a four-year in fall after high school (1.6 pp)
Survey Sample
Surveys conducted in spring 2014
◦ All treatment schools (36 + additional randomized group from 2013)
◦ 42 control schools
Treatment/Control Balance
◦ Less representation among Hispanic students (as in the administrative data)
Surveys conducted in spring 2014
◦ All treatment schools (36 + additional randomized group from 2013)
◦ 42 control schools
Preliminary Survey Results
Key Differences
◦
◦
◦
◦
◦
◦
◦
◦
Degree expectations improve
Plans to work full- or part-time after college or to enter the military decrease
Similarly likely to participate in “soft” college prep activities (college visits, website, test prep)
More applications submitted and greater likelihood of receiving help
More likely to attempt AP exams or to retake college entrance exams
More likely to complete FAFSA (self-reported)
More likely to have submitted a deposit
Less likely to have talked to a high school counselor or teacher about admissions
Discussion
Unsurprising that treatment effects are small at the school level
Advisers likely spend time with inframarginal students who are already going to college as
opposed to targeting support at students who need it most
Degree of complementarity is potentially important part of the story
Cost-benefit analysis
Scalability
Future work
◦ Will examine persistence outcomes to observe whether advisers have effect on “fit”