Transcript Reducing Barriers to Postsecondary Education Through Innovative Interventions: Evidence from Canadian Field Experiments"

Preliminary: not for quotation
***LILE and Aspirations question
Reducing Barriers to Postsecondary Education
Through Innovative Interventions: Evidence from
Canadian Field Experiments
McGill University Social Statistics Speaker Series
March 21, 2012
Marc Frenette
1
Reducing Barriers to Postsecondary: The Need
for Hard Evidence
 Improving access to PS important for many reasons:
• Private gains (Card, 1999; many others..)
• Societal gains, including productivity enhancement (Coulombe
and Tremblay, 2006), crime reduction (Lochner and Moretti,
2004), longer life expectancy (Lleras-Muney, 2005), improved
health (Arendt, 2008), and civic participation (Milligan, Moretti,
and Oreopoulos, 2004)
• Equalizing opportunities / increasing intergenerational mobility
2
Reducing Barriers to Postsecondary: The Need
for Hard Evidence
 The Canadian literature has successfully identified several
important barriers to PS:
•
•
•
•
3
Income (Frenette, 2008)
Financial literacy (Frenette and Robson, 2011)
Academic performance (Frenette, 2008)
Career information (Frenette, 2009)
Reducing Barriers to Postsecondary: The Need
for Hard Evidence
 Although all four barriers are amenable to policy intervention,
we have little to no hard evidence on best practices in
Canada:
• Students who qualify for grants or participate in academic /
career counseling are likely a select group
• Lack of natural experiments in Canada
• Field experiments, using random assignment, are ideal, but
rare…
4
The Canadian Millennium Scholarship Foundation
Recognized the Need for Hard Evidence
(and they had hard cash!!!)
 Commissioned the Social Research and Demonstration
Corporation (SRDC) to design, implement, and evaluate
three interventions to reduce specific barriers to PSE:
• BC AVID
– Academic intervention for middle-achieving students
• Learning Accounts (Future to Discover)
– Early promise of a large non-repayable grant for low-income
students
5
• Explore Your Horizons (Future to Discover)
– Career / education information
AVID
 Advancement Via Individual Determination (AVID)
 Serves 400,000 students in 4,500 schools worldwide
 Goal: raise academic performance, and subsequently,
university attendance
 Targets students on the margin
• Mostly Bs and Cs (middle-achieving)
• No extreme behaviour issues
• Desire to attend university
6
AVID
 What is AVID?
• Academic intervention
• Elective class (replacing a regular elective) offered throughout
high school
• Consists of:
– Curriculum studies (40%)
– Tutorials assisted by local college students (40%)
– Motivational activities regarding PS (20%)
7
AVID
 Mechanisms through which AVID may help students (Dunn et
al., 2008)?
• Study skills (time management, note taking, etc.)
• “Untracking” students (students choosing advanced courses)
• Mentoring effects (continued contact with AVID teacher and
tutors)
8
• Peer effects (continued contact with students sharing similar
characteristics)
AVID
 Can AVID make a difference?
• Cognitive skills are not malleable after age 14 (Heckman, 1995),
when AVID begins
– But AVID helps students use their existing cognitive skills more
efficiently by helping them become better learners
• Furthermore, non-cognitive skills (motivation, self-discipline) are
controlled by the prefrontal cortex, which is malleable until late
adolescence (Heckman, 2000; Shonkoff and Phillips, 2000)
9
AVID
 Non-experimental evidence on AVID’s effectiveness:
• Mehan et al. (1996):
– Improvements in college participation
• Watt et al. (2006)
– AVID districts in Texas saw gains in graduation rates, advanced course
enrollments, and international baccalaureate testing
• Watt et al. (2007)
– AVID participants had higher aspirations, knowledge about college, and
academic preparation compared to peers
• Nagaoka and LaForce (2010)
– Propensity score matching study in Chicago
– Small improvements in English and Math in grade 9; fewer absences
 All suffer from selection bias
10
• Those most likely to benefit from AVID will sign up
BC AVID
 First (and only) scientific evaluation of AVID
 1,241 students recruited at 14 BC high schools*
• Two cohorts (2005 and 2006) followed throughout HS and until
university age
• Administrative and survey data used for follow-up
 ‘AVID eligible’ students recruited through rigorous process:
• Middle-achieving (mostly Bs and Cs)
• No extreme behaviour issues
• Desire to attend university
*Schools and students had to apply to participate and waiting lists were created when classes reached
the limit of 30 (just like the real AVID). Informed consent was also required from the parents to collect data.
11
Evaluation approach
 Random assignment within high schools
 Impact:
• Treatment group outcome – Control group outcome
• Controlling for baseline characteristics
12
Focus of this study
 Impact of the offer of AVID on highest level of education
aspired*:
• Early demand indicator
 Future work will report on university / college application
/ attendance
* “No matter what you plan to be doing in a year from now (grade 12), what is the highest level of education you would like to get?”
13
Large impact on university aspirations for boys
and first-generation students
Table 2: Impact of offer of AVID on probability of aspiring to university
Outcomes (%)
Treatment
Control
All
Boys
55.31
Weaker AVID skills
to begin with?
51.09
Impact
4.23
(3.10)
53.31
45.68
7.62 *
(4.41)
Girls
57.22
55.19
2.03
(4.33)
No parent with postsecondary
52.09
43.32
8.77 **
(4.45)
Parent with postsecondary
60.62
61.56
-0.94
(4.68)
Note: Standard errors are in parentheses. Significant impacts are denoted by '***'
(1%), '**' (5%), and '*' (10%).
14
No impact on non-university PS, as expected
Table 3: Impact of offer of AVID on probability of aspiring to non-university
postsecondary
Outcomes (%)
Treatment
Control
All
39.37
41.86
Impact
-2.49
(3.17)
Boys
42.86
44.78
-1.92
(4.74)
Girls
36.09
40.01
-3.93
(4.11)
No parent with postsecondary
42.79
47.57
-4.79
(4.56)
Parent with postsecondary
34.14
33.45
0.69
(4.61)
15
Note: Standard errors are in parentheses. Significant impacts are denoted by '***'
(1%), '**' (5%), and '*' (10%).
AVID may need more time to work…
Table 5: Impact of offer of AVID on probability of obtaining 80% or more on grade 10 and 12 English course marks and on
Grade 10 course mark
Outcomes (%)
Treatment Control
Impact
All
23.00
24.02
-1.02
Grade 12 course mark
Outcomes (%)
Treatment Control
Impact
20.99
18.88
(2.38)
Boys
15.37
16.43
-1.06
29.70
30.54
-0.84
(3.62)
11.29
9.27
(2.10)
16.98
12.48
(3.29)
Girls
2.11
Grade 12 final mark
Outcomes (%)
Treatment Control
Impact
4.51
(1.59)
9.05
6.43
(3.07)
24.48
24.36
0.12
(3.39)
2.03
2.62
(2.41)
13.42
11.40
2.03
(2.73)
No parent with
19.96
20.77
-0.81
16.34
9.22
7.13 ***
9.31
4.78
4.53 **
postsecondary
(3.10)
(2.52)
(1.88)
Parent with
26.82
27.84
-1.01
27.71
30.79
-3.09
13.74
15.33
-1.59
postsecondary
(3.84)
(4.19)
(3.29)
Note: Standard errors are in parentheses. Significant impacts are denoted by '***' (1%), '**' (5%), and '*' (10%). The
grade 12 English final is a weighted average of the grade 12 English course mark (80%) and the grade 12 English
16
Potential biases/evaluation challenges
Those we can rule out

Program take-up
•

Spillover of AVID techniques
•

17
Very minor issue based on earlier comparison with similar non-AVID schools (Cornell notes)
Attrition bias (survey data analysis only)
•
•

96.7% accepted AVID offer
19.6% (treatment) vs 23.8% (control) between grade 9 and 12
No important changes in baseline characteristics
Teacher grading bias
•
•
AVID students are well known
Argument does not apply to differences in sub-group impacts
Potential biases/evaluation challenges
Those we can not rule out

Program drop-out
•
•

Substitution bias (displacing non-experimental treatment)
•

Earlier report found reverse: course load was more challenging among treatment group
Timing
•
•
18
37.6% withdrew by grade 12 (mostly in that year: wanted to pick other courses to apply
skills)
Estimated impacts = Intention-to-treat effects
•
Lagged effect
Students coped with more difficult course load as they were only beginning to learn AVID
skills
AVID now being run before HS in some jurisdictions
Potential biases/evaluation challenges
Those we can not rule out

Reactions to inequity caused by randomization
•
•

Sample size
•
19
Positive or negative impact on control group
Currently following outcomes of similar students in matched non-AVID schools
Affects statistical significance (especially in sub-group analysis)
BC AVID Final Report
 Final report (including cost-benefit study) in 2013
20
Future to Discover Overview
 Two interventions designed to increased PSE access:
• Learning Accounts
– Early promise of non-repayable grant (up to $8,000) for low-income youth
– Offered in NB
• Explore Your Horizons
– Career and PSE planning intervention
– Several components (workshops, magazines, parental outreach, etc.)
– Offered in NB and MB
21
How Can Learning Accounts Help?
 Reduces cost of post-secondary, like other grants (clawedback from loans)
 Potentially addresses several design issues with existing
grants:
– Students are informed of the grant
– Early promise of aid
– No obligation to apply for loans
o May encourage loan averse students to seek grant
 Palameta and Voyer (2010)
 Lab experiment: 5-20% students rejected grants when couple with a loan
offer, but accepted same grant without loan offer (even though loan could
be forfeited without compromising grant)
o With no loans, Learning Accounts represents additional aid
22
How Can Explore Your Horizons Help?
 Informs students of costs and benefits of PSE
• Financial literacy of students very poor regarding PSE (Frenette
and Robson, 2011)
 Helps students understand educational requirements for the
careers they wish to pursue
• Knowledge of educational requirements an issue for many
students (Frenette, 2009)
23
Methodology
 Students are randomly assigned to receive the intervention
(treatment group) or not (control group)
 Three possible interventions:
• LA
• EYH
• LA + EYH
24
Table 1: LA IMPACTS ON PSE ENROLMENT
New Brunswick
Fr-LA-Eligible
LA
En-LA-Eligible
Comparison Impact
Group
Group
(s.e)
76.34
66.07
75.20
61.40
Parents with High school or less (FGF)
67.33
55.28
Parents with any PSE
88.25
75.68
Boys
71.20
58.60
Girls
82.74
70.81
10.27
(4.30)
13.80
(4.92)
12.04
(6.75)
12.57
(5.25)
12.60
(6.85)
11.93
(5.33)
247
262
LA
Comparison Impact
Group
Group
67.69
61.00
*** 65.70
56.98
*
60.87
55.11
**
74.12
65.82
*
60.98
45.48
**
73.18
72.25
240
255
(s.e)
Enrolled in PSE institution (%)
ALL
LILE
Sample size
No PSE credential
**
S ource: FTD 66 month survey, FTD 66 month proxy survey, FTD Administrative data
Notes: Estimates regression adjusted.
Sample sizes vary for individual measures because of missing values.
Statistical significance levels are indicated as * = 10 per cent; ** = 5 per cent; *** = 1 per cent.
Rounding may cause slight discrepancies in sums and differences.
25
6.68
(4.21)
8.71 *
(4.69)
5.76
(6.54)
8.30
(6.06)
15.50 **
(7.84)
0.93
(5.24)
Table 2: LA IMPACTS ON PSE ENROLMENT BY TYPE OF INSTITUTION
New Brunswick
Fr-LA-Eligible
LA
En-LA-Eligible
Comparison Impact
LA
Comparison Impact
Group
Group
(s.e)
Group
Group
(s.e)
ENROLLED IN UNIVERSITY
ALL
29.42
25.75
25.55
26.28
LILE
25.35
17.74
22.45
23.04
Parents with High school or less (FGF)
22.52
16.52
24.51
18.11
Parents with any PSE
37.99
34.42
25.75
35.21
Boys
16.81
18.03
12.21
13.71
Girls
39.59
33.35
3.68
(4.08)
7.61 *
(4.36)
6.01
(5.16)
3.56
(6.54)
-1.22
(5.36)
6.24
(5.37)
36.97
34.37
-0.73
(3.96)
-0.59
(4.41)
6.40
(5.48)
-9.46
(5.92)
-1.50
(4.95)
2.60
(5.77)
ENROLLED IN COLLEGE
ALL
49.20
36.56
32.68
25.48
LILE
51.48
35.42
*** 31.96
24.50
Parents with High school or less (FGF)
46.18
28.79
*** 28.34
26.61
Parents with any PSE
54.51
43.27
35.70
24.74
Boys
57.18
35.97
*** 34.25
25.43
Girls
44.66
34.96
12.64
(4.98)
16.07
(5.28)
17.39
(6.22)
11.24
(7.84)
21.21
(7.59)
9.70
(6.44)
29.56
27.34
26
**
7.20 *
(3.96)
7.46 *
(4.16)
1.73
(6.52)
10.96 *
(5.90)
8.82
(6.71)
2.21
(5.87)
Why did Learning Accounts work in Francophone
sector?
 Possible explanation: supply constraints
 Data from New Brunswick government:
• Anglophone programs were oversubscribed
• Francophone programs were undersubscribed
27
LA IMPACTS ON PSE APPLICATIONS BY TYPE OF INSTITUTION
New Brunswick
Fr-LA-Eligible
LA
En-LA-Eligible
Comparison Impact
LA
Comparison Impact
Group
Group
(s.e)
Group
Group
APPLIED TO UNIVERSITY
ALL
32.08
25.31
30.22
28.18
LILE
28.91
17.51
26.69
24.72
Parents with High school or less (FGF)
23.78
15.72
28.32
16.45
Parents with any PSE
41.80
34.75
32.45
39.16
Boys
19.68
17.71
16.66
15.26
Girls
42.29
33.02
6.77 *
(4.08)
11.40 **
(4.44)
8.06
(5.17)
7.05
(6.61)
1.98
(5.43)
9.26 *
(5.50)
40.26
38.20
APPLIED TO COLLEGE
ALL
48.89
35.26
*** 41.64
27.93
LILE
51.92
33.21
*** 42.80
28.55
Parents with High school or less (FGF)
48.87
25.82
*** 40.51
29.10
Parents with any PSE
50.04
44.88
41.88
27.42
Boys
56.89
34.90
*** 43.04
24.08
Girls
43.25
34.38
40.67
30.72
28
13.63
(4.64)
18.71
(5.16)
23.05
(6.25)
5.16
(7.54)
21.99
(7.74)
8.87
(6.31)
(s.e)
2.04
(3.98)
1.97
(4.58)
11.87 **
(5.33)
-6.71
(5.94)
1.40
(6.08)
2.06
(6.12)
13.71
(4.21)
14.26
(4.71)
11.40
(7.35)
14.46
(6.27)
18.96
(6.88)
9.95
(6.52)
***
***
**
***
Table 3: EYH IMPACTS ON PSE ENROLMENT
New Brunswick
Manitoba
Francophone
Anglophone
EYH
Comparison
Impact
EYH
Comparison Impact
Group
Group
(s.e)
Group
Group
(s.e)
ALL
72.62
67.82
81.73
77.54
LILE
61.43
52.88
74.01
60.63
NOT LILE
76.32
73.78
83.81
88.13
Parents with High school or less (FGF)
62.90
55.64
72.38
59.42
Parents with any PSE
76.83
72.46
84.99
86.95
Boys
66.68
57.53
77.64
71.55
Girls
78.66
77.53
86.68
81.81
Aboriginal
63.72
61.67
.
.
4.19
(2.31)
13.38
(5.01)
-4.32
(2.63)
12.96
(5.01)
-1.97
(2.47)
6.09
(3.96)
4.86
(2.87)
.
.
478
395
484
677
EYH
Comparison
Impact
Group
Group
(s.e)
74.10
70.26
*** 63.87
55.52
80.42
79.69
*** 58.98
56.15
80.87
76.76
67.69
60.70
80.29
79.20
.
.
3.85
(2.63)
8.35
(4.66)
0.74
(3.20)
2.83
(5.54)
4.11
(2.73)
6.99
(4.06)
1.08
(3.21)
.
.
471
646
Enrolled in PSE institution (%)
Sample size
4.80
(3.04)
8.55
(6.89)
2.54
(3.52)
7.27
(6.27)
4.36
(3.45)
9.15 *
(4.86)
1.12
(3.82)
2.06
(11.13)
S ource: FTD 66 month survey, FTD 66 month proxy surveyFTD Administrative data.
Notes: Estimates regression adjusted.
Sample sizes vary for individual measures because of missing values.
Statistical significance levels are indicated as * = 10 per cent; ** = 5 per cent; *** = 1 per cent.
Rounding may cause slight discrepancies in sums and differences.
29
*
*
*
*
Table 4: EYH IMPACTS ON PSE ENROLMENT BY TYPE OF INSTITUTION
New Brunswick
Manitoba
Compariso
n
Group
Impact
EYH
Group
(s.e)
Group
ENROLLED IN UNIVERSITY
ALL
51.33
47.40
48.74
43.78
LILE
34.55
33.78
33.05
19.10
NOT LILE
57.41
53.48
55.92
58.23
Parents with High school or less (FGF)
34.69
32.81
29.86
22.87
Parents with any PSE
57.94
53.42
57.33
54.07
Boys
42.31
35.45
38.14
34.81
Girls
60.40
58.87
58.69
51.42
Aboriginal
35.87
36.41
3.93
(3.20)
0.77
(6.81)
3.93
(3.79)
1.87
(5.85)
4.52
(3.94)
6.86
(4.63)
1.53
(4.25)
-0.54
(12.24)
.
.
ENROLLED IN COLLEGE
ALL
25.23
20.33
38.47
36.97
LILE
25.70
15.17
41.19
34.73
NOT LILE
24.84
21.72
36.10
38.90
Parents with High school or less (FGF)
29.76
15.42
38.77
30.59
Parents with any PSE
23.60
21.76
37.62
40.36
Boys
24.63
17.87
46.65
38.44
Girls
26.35
22.16
32.48
34.30
Aboriginal
29.53
23.89
4.89
(3.00)
10.53 *
(5.92)
3.12
(3.55)
14.35 **
(5.97)
1.84
(3.32)
6.76
(4.14)
4.19
(4.40)
5.64
(11.74)
.
.
EYH
30
Francophone
Compariso Impact
n
Group
(s.e)
Anglophone
Compariso
n
Group
Impact
Group
4.97 *
(2.71)
13.95 ***
(4.28)
-2.31
(3.46)
7.00
(4.61)
3.26
(3.47)
3.33
(4.16)
7.26 *
(3.71)
.
.
43.13
41.04
29.76
22.01
49.51
53.05
17.54
20.92
53.81
50.25
34.02
29.95
51.54
51.49
.
.
2.08
(2.51)
7.74
(4.43)
-3.54
(3.62)
-3.38
(4.20)
3.56
(3.25)
4.07
(3.93)
0.04
(3.82)
.
.
1.50
(3.00)
6.46
(5.42)
-2.80
(3.38)
8.18
(5.31)
-2.74
(3.48)
8.21 *
(4.65)
-1.82
(3.79)
.
.
27.57
24.19
25.50
24.89
29.84
24.04
29.75
26.80
26.87
23.15
28.72
26.95
26.33
21.78
.
.
EYH
(s.e)
3.38
(2.63)
0.61
(4.47)
5.80
(3.30)
2.94
(5.78)
3.72
(3.12)
1.77
(4.11)
4.55
(3.75)
.
.
*
*
Table 5: EYHLA IMPACTS ON PSE ENROLMENT
New Brunswick
Fr-LA-Eligible
En-LA-Eligible
EYHLA Comparison Impact
EYHLA Comparison Impact
Group
Group
(s.e)
Group
Group
ALL
72.60
66.23
69.41
59.41
LILE
72.14
60.62
66.52
55.01
Parents with High school or less (FGF)
70.86
56.72
60.89
53.77
Parents with any PSE
75.29
76.14
75.67
67.01
Boys
62.42
59.78
67.88
41.75
Girls
84.52
70.11
6.37
(4.42)
11.52 **
(4.97)
14.15 **
(6.81)
-0.85
(6.06)
2.64
(7.35)
14.41 **
(5.63)
70.41
73.10
244
262
237
255
(s.e)
Enrolled in PSE institution (%)
Sample size
S ource: FTD 66 month survey, FTD 66 month proxy survey, FTD Administrative data
Notes: Estimates regression adjusted.
Sample sizes vary for individual measures because of missing values.
Statistical significance levels are indicated as * = 10 per cent; ** = 5 per cent; *** = 1 per cent.
Rounding may cause slight discrepancies in sums and differences.
31
10.00 ***
(3.80)
11.50 **
(4.51)
7.11
(6.54)
8.66
(5.27)
26.13 ***
(6.39)
-2.69
(5.44)
Table 6: EYHLA IMPACTS ON PSE ENROLMENT BY TYPE OF INSTITUTION
New Brunswick
Fr-LA-Eligible
En-LA-Eligible
EYHLA Comparison Impact
EYHLA Comparison Impact
Group
Group
(s.e)
Group
Group
ENROLLED IN UNIVERSITY
ALL
31.89
24.30
32.32
25.54
LILE
26.32
16.69
28.97
21.63
Parents with High school or less (FGF)
24.66
16.79
23.35
16.23
Parents with any PSE
38.22
33.68
41.14
34.58
Boys
22.48
17.28
26.79
13.64
Girls
40.04
31.54
7.59 *
(4.03)
9.63 **
(3.98)
7.88 *
(4.49)
4.54
(6.30)
5.20
(5.63)
8.49
(5.67)
39.25
33.55
ENROLLED IN COLLEGE
ALL
42.52
38.17
24.79
24.99
LILE
43.30
36.58
26.30
24.62
Parents with High school or less (FGF)
42.38
31.37
24.94
27.83
Parents with any PSE
45.70
43.00
22.26
24.39
Boys
40.29
37.64
30.29
22.06
Girls
46.21
37.25
4.34
(5.15)
6.72
(5.61)
11.01
(7.33)
2.70
(7.09)
2.64
(7.76)
8.96
(6.83)
17.78
28.04
32
(s.e)
6.78 *
(3.76)
7.34 *
(4.09)
7.12
(4.85)
6.56
(5.92)
13.15 **
(5.43)
5.71
(5.40)
-0.20
(3.78)
1.68
(4.15)
-2.88
(5.93)
-2.14
(5.94)
8.23
(6.38)
-10.26
(6.26)
Potential biases/evaluation challenges

Program take-up/drop-out
•
•

Spillover effects
•
•

33
Not an issue for LA
EYH workshops held after class
Substitution bias (displacing non-experimental treatment)
•
•

Everyone received offer of LA, but about 30% did not remember a couple of years later
Attendance in EYH workshops was poor, especially in Manitoba
– Non-mandatory, after school
Minor impact in LA (small reduction in non-repayable aid for some groups)
EYH: looking into it
Attrition bias
•
Enrolment numbers use administrative data
Potential biases/evaluation challenges

Reactions to inequity caused by randomization
•
•

Sample size
•
34
Positive or negative impact on control group
No mechanism for testing this (did not follow non-experimental schools)
Affects statistical significance (especially in sub-group analysis)
Future to Discover Final Report
 Final report (including cost-benefit study) in September
2012
35
Appendix: Related academic interventions

QOP (Hahn et al., 1994; Rodriguez-Planas, 2010)
•

IHAD (Kahne and Bailey, 1999)
•

Includes financial incentives
Includes early promise of ‘last dollar scholarship’
Career Academies (Kemple and Willner, 2008)
•
Focused on career themes
Treatment broadly
similar to AVID, but
focus on
low-income students

GEAR UP

Upward Bound (Seftor et al., 2009)

Upward Bound Math-Science (Seftor and Calcagno, 2010)
36