“Improving the Success of Students at Risk of Dropping Out: Changing Practices and Policies” Dr. Andrew Parkin Associate Executive Director Canada Millennium Scholarship Foundation Canada 18 th Annual 2009 European Access Network Annual Conference

Changing the Culture of Campus Towards an Inclusive Higher Education

June 22-24, 2009 York, UK

Outline

1.

Introduction to the issue of persistence 2.

Who leaves PSE and why?

3.

a) b)

Strategies to improve persistence

Foundations for Success LE,NONET

4.

Reflections on experimental research

Introduction to the Issue of Persistence

Introduction to the Issue of Persistence

• Persistence: – the ability of students to continue their post-secondary studies from one year to the next and ultimately complete their programs • Access  Success – Meaningful access requires that students brought into to PSE must be successful in their studies • Low levels of persistence pose a problem... – For students: left without the credential they need – For institutions: poor use of resources and poor performance – For societies: • lower educational attainment • exacerbates social cleavages

An Increasingly Important Concern

As a recent OECD report puts it: • “the growing portion of disadvantaged students enrolled in tertiary education makes the ongoing issue of their retention and programme completion an increasingly important concern in tertiary education.”* * Santiago, Paulo, Karine Tremblay, Ester Basri and Elena Arnal. 2008.

Tertiary Education for the Knowledge Society

, Volume 2. Paris: OECD.

Persistence: How Much Do We Know?

• “We know very little about how many students drop out of programs, or why” (Rae, 2005) • Little research on the issue of persistence in PSE in Canada until recently – Previous data on persistence rates limited to institution-specific studies – Little program evaluation or evidence of what works – New research now becoming available: • Canada Millennium Foundation pilot projects and program evaluations

The Need for Program Evaluation

“Presently…there is little evidence about the effects of institutional support programmes on student outcomes.” Paulo Santiago, Karine Tremblay, Ester Basri and Elena Arnal. 2008.

Tertiary Education for the Knowledge Society

,

Volume 2

. Paris: OECD.

“It is ironic that institutions that spend so much time and money insisting on evidence-based decisions, spend so little time on research that evaluates higher education itself. Research on the postsecondary sector is inadequate and poorly publicized.

This should change.” Bob Rae,

Ontario: A Leader in Learning. Report and Recommendations.

Toronto: Government of Ontario, Ministry of Training, Colleges and Universities (2005)

The Challenges of Program Evaluation • Student support programs are notoriously difficult to evaluate because:

– Programs are designed and implemented without the requirements for a viable evaluation in mind – Data that would facilitate evaluation are not available – It is difficult to isolate the effects of the program from the possible effects of external factors – There is no counterfactual (no reasonable “control” group against which to compare the outcomes of program recipients)

Who Leaves PSE and Why?

Persistence Rates in Canada

Overall Persistence Rates in Post-Secondary Education in Canada Graduated (%) Still in post secondary education (%) Discontinued post secondary education (%) College

Year 1 Year 2 Year3 Year 4 Year 5

University

Year 1 Year 2 Year3 Year 4 Year 5 12.0

36.9

57.0

66.2

73.1

1.1

3.6

11.2

45.0

69.4

75.2

45.8

25.1

14.8

8.8

91.0

86.7

78.8

45.2

20.4

12.9

17.3

17.9

19.0

18.0

7.9

9.6

9.9

9.8

10.2

Source: Finnie and Qiu, Table 6b.

Barriers to Access and Persistence

15% 10% 5% % 35% 33% 30% 25% 20% 22% Finances 31% 14% Career indecision 13% 29% Lack of interest 27% Never attended 19% 9% Discontinued 8% 12% 11% 10% Program not what expected Employment Academic challenges Personal/family Malatest (2007),

Class of 2003

Who Leaves PSE and Why?

Self-Reported Reasons for Discontinuing Post-Secondary Studies Class of 2003 Study YITS

Lack of interest/ lack of program fit/ lack of career direction Financial reasons (other than desire to work) Academic reasons Desire to work 52% 23% 14% 11% 32% 11% 6% 7%

Who Leaves PSE and Why?

Factors correlated with low persistence: • Poor academic performance (both secondary and PSE levels) • Low engagement • Inadequate financial aid package or high levels of debt • Uncertainty about career goals • Lower levels of parental education (in some studies) • Aboriginal ancestry • Gender (men are more likely to drop out than women) • Age & family status (older students and students with dependent children are more likely to drop out)

Percent of PSE Students Aged 24-26 Who Had Discontinued Their Original Stream of PSE *, By Grade Average in High School

15 10 5 0 40 35 30 25 20 38 59% and less 34 30 60% -69% 27 22 70% -79% College Stream University Stream 18 14 11 7

* Note: while some of these students will have discontinued their studies, others will have switched streams.

80% -89% 90% -100%

Source: Shaienks, Danielle, Tomasz Gluszynski and Justin Bayard. 2008.

Postsecondary Education: Participation and Dropping Out: Differences Across University, College and Other Types of Postsecondary Institutions.

Ottawa: Statistics Canada.

Percent of Post-Secondary Students who Have Discontinued Their Studies (By Age Group)

25 20 21 23 15 16 14 14 10 9 5 0 18-20 20-22 22-24

Source: YITS (Cohort B) – special calculation.

* Excludes First Nations youth living on reserve

15 20 24-26 Non-Aboriginal Students Aboriginal Students*

Who Leaves PSE and Why?

• Those who persist are more likely to attempt more than one program than are those who drop out • A key difference between those who persist and those who leave is the ability to make adjustments along the way • Importance of “resilience” (the capacity of overcome obstacles, adapt to change, or to survive and thrive despite adversity) – Those who persist are able to make adjustments that help them to stay enrolled – Factors contributing to resilience in youth include supportive relationships with adults and parental expectations

Strategies to Improve Persistence

Foundations for Success LE,NONET

Research on Policy Responses

• Persistence should be addressed through a comprehensive approach – Need to address interconnected barriers to success – Need to work at institutional level rather than isolated policies implemented by various departments • There are many programs designed to improve persistence, but few of these are evaluated for impact • Student support programs can be implemented in the context of a research endeavour that allows the impact of the program to be assessed effectively – Confirms that programs deliver the intended benefits to students – Confirms that resources are well-spent

Overcoming Barriers to Access and Success (Millennium Pilot Projects)

Career development activities Financial incentives Community Support Academic preparation Improving access to PSE Academic support Career development Financial support Persistence GRADUATION Mentoring & cultural support Off-campus community support

Tackling the Drop-Out Rates in Community Colleges through Financial Incentives and Case Management

Foundations for Success

Research questions: – Do case manager-mediated support services increase the probability of completing a college program?

– Do financial incentives in combination with case manager-mediated support services increase the probability of completing a college program?

Project Participation: At-Risk Factors

1) Remedial English/Communications:

•

Measure

: English Placement assessment results (slight variation within participating colleges) •

Intervention

: students are either streamed to regular Communications or developmental course and are encouraged to undertake tutoring and related academic support

2) Mentoring:

•

Measure

: Fast-Track survey question on whether the student self identified as someone who would benefit from a mentor.

•

Intervention

: Assignment of a mentor.

3) Career indecision:

• •

Measure

: Four Fast-Track career clarity questions, with a cumulative score greater than 10 (i.e. neutral response to strongly agree/disagree).

Intervention

semesters) : two “Career Gear” workshops; Myers-Briggs; Strong Campbell; one group debriefing; one one-on-one with counsellor (over 2

Innovative Features of the Foundations for Success Model

• Post-admissions testing where responses result in actual redirection to existing services • Case management approach to advise identified at-risk students (one-on-one advisement) • “Case managers” follow students’ progress for two years. This involves encouragement, identification of students’ needs and challenges, and redirection to appropriate services.

Random Assignment

New students in 2-year programs are invited to complete the Fast Track survey, language assessment and sign informed consent, making them eligible for the study Students deemed at risk on at least one item Students not deemed at risk (no further involvement in project) Random Assignment occurs Program Group 1 SERVICES

Academic, mentorship and career exploration support

Program Group 2 SERVICES PLUS

Academic, mentorship and career exploration support

and

financial incentives

Program Group 3 COMPARISON

Number of Participants per College

Services Plus Compensation

PROJECT TOTAL Seneca College

404 406 387

1197 Mohawk College Confederation College TOTAL

393 242

1039

393 434 242 242

1041 1062 1219 726 3142

Year 1 Results Participation in FFS Activities

• Service group significantly more likely to participate in any FFS activities compared to Control group • Service Plus group significantly more likely than Service group to participate in any FFS activities Source: 2008 College Administration Data, n=2,008.

Participation – Sub-group Analysis

• Higher Semester 1 participation rates among ESL students, first generation, older, low-income and low-confidence students.

• Higher Semester 2 participation rates among ESL students, older students, women, low income and low-confidence students.

61% 43% 70% 51% 63% 45% 58% 40% 63% 43% 82% 64% 73% 55% 58% 38% 70% 54% 59% 37% 59% 36% Semester 1 Semester 2

Semester 1 GPA

• Service Plus group GPA was higher (2.13) compared to the Control group (2.01) 2.01

2.06

2.13

Control group Service Group Service Plus Group* Source: 2008 College Administration Data, n=2,008, missing GPAs were imputed using a regression model.

Semester 2 GPA

• Service Plus group had a higher GPA in Semester 2 (1.96) compared to the Control group (1.79).

1.79

1.82

1.96

Control group Service Group Service Plus Group** Source: 2008 College Administration Data, n=2,008, missing GPAs were imputed using a regression model.

Student Retention – Beginning of Year 2

• Service Plus group significantly more likely to still be in program (67.2%) compared to Control group (62.6%).

• Adjusting for the students that did not participate in any FFS activities, the adjusted effect of the Service Plus treatment is a 6.4% increase in retention one year into students’ college program.

65.8% 67.2% 62.6% Control group Service Group Source: 2008 College Administration Data, n=1,711.

Service Plus Group*

Sub-Group Analysis – Who Gets the Most Benefit from the Interventions?

•

Retention:

Service Plus ESL students, low-income students, and students with high school grades of 65 or lower show significantly higher retention rates than similar students in the Control group.

60% 74% 55% 73% 44% 59% ESL students Family Income Less than $25,000 Control group High School Grades of Service Plus Group 65 or Lower Source: 2008 College Administration Data, n=1,711, Cohort 1 only.

“Success after enduring many hardships” (Salish)

Improving the persistence rate of Aboriginal learners at the University of Victoria

LE,NONET

• Pilot project to test the effectiveness of initiatives to improve the retention of Aboriginal students at the University of Victoria • Project recognizes that improved outcomes necessitates change not only in students but also in the university – “the onus for adjustment” should not be placed solely on Aboriginal students • LE,NONET project involves the creation and implementation of a series of programs and support structures: – Student Mentoring – Bursaries (regular and emergency) – Community Internship – Research Apprenticeship – Staff and Faculty Aboriginal Cultural Training (SFACT)

LE,NONET

Research question: • will a series of interventions involving financial, academic, peer and cultural support have a demonstrable effect on Aboriginal students’ likelihood of completing their program of studies?

• is any particular type of intervention more effective and do the interventions have greater impact when delivered in combination with one another?

The Research Framework

• Research will document the process through which changes are proposed, developed, implemented, and modified at the university, faculty, department and program level over the course of the project • Since one of the objectives of the program is to change the university culture, it is possible for the program to affect non-participants as well as participants (cannot use random assignment) – evaluate the direct effects of participation in the various programs offered by the LE,NONET project – evaluate the indirect effects that such programs might have on students who decline to participate • Comparison of the persistence of Aboriginal students during the periods before and after the program was introduced • The LE,NONET approach will allow students to determine the depth of their involvement in the programs — making differing ‘levels of exposure’ to the interventions a key variable of interest.

Participation

• Number of participants (2006 & 2007): 139 – Bursaries = 87 – Mentor = 17 – Mentored = 35 – Preparation seminar = 44 – Community intership = 24 – Research apprenticeship = 17 • Historical cohort: 997 • Non-participants: 728

Interim Findings

• The majority of participants (77%) reported that their participation in LE,NONET program components contributed to the development of their sense of self as an Aboriginal person. • A strong majority (87%) agreed that their participation in the programs contributed to their sense of connection to the on-campus Aboriginal community. • Just over half (54%) said that their involvement with LE,NONET also contributed to their sense of belonging at the university.

Interim Findings: Return to School

• Overall, about half of the research participants said that LE,NONET had contributed to their decision to return to school the following year.

• For some students, the financial support they received through the Project made it financially viable for them to afford their tuition and living expenses.

• Other respondents said that returning to school was made easier by the emotional support and sense of community provided by LE,NONET.

Registration Status (Interim): LE,NONET

70% 60% 50% 40% 30% 20% 10% 0% Continuing Graduated Withdrawn Historical Cohort Participants Non-Participants

Reflections on Experimental Research

Reflections on Experimental Research

1.

2.

3.

These projects show that student support programs can be implemented in the context of experimental research initiatives that will lead to an increase stock of evidence about what may or may not work.

Experimental research is possible but not always easy to put in place. Obstacles include: a) b) c) d) e) Money Time Confusion about the objectives of the research • • Tensions between educators and researchers Sticking to the script What is the measure of success?

Recruitment of participants All of these obstacles can be overcome, but only when researchers develop effective partnerships with educators and administrators.

Andrew Parkin [email protected]

Canada Millennium Scholarship Foundation 1000 Sherbrooke West, Suite 800 Montreal, QC H3A 3R2 1-877-786-3999 All our publications are available online: www.millenniumscholarships.ca