Transcript Slide 1

Building Evidence that Counts:
Evaluating Career Pathways
Interventions for Disconnected
Youth and Adults in the U.S.
David Fein
Abt Associates Inc
[email protected]
Improving Education through Accountability and
Evaluation: Lessons from Around the World
Rome, Italy
3 October, 2012
Acknowledgments
 Work on Innovative Strategies for Increasing Self-Sufficiency (ISIS)
evaluation sponsored by Administration for Children and Families,
U.S. Department of Health and Human Services with major support
from the Open Society Foundations, additional support from Joyce
and Kresge foundations
 Presentation represents the author’s views only
 Paper available at: http://www.projectisis.org/wpcontent/uploads/2012/05/ISIS-Career-Pathways-Framework-OPRE2012-30_5-16-12.pdf
2
Introduction



Large population of adults and youth in the U.S. with low skills, little/no postsecondary education, and poor prospects for self-sufficiency
In the 1990s-2000s, national policies emphasized labor force attachment
(versus education and training)
Evidence from social experiments influenced this emphasis
– Consistent modest positive findings for employment-focused interventions;
disappointing findings for education and training
– Some argued human capital investments should be limited to younger ages

Findings characterize traditional educational approaches not well-suited to
second-chance populations
3
The Ground Is Shifting






Increasing evidence of limits to strictly employment-focused strategies,
especially in a weak economy
Recession hits low-skilled jobs hardest and accelerates technological shifts
widening skills premiums
Improved understanding of challenges facing individuals seeking post2ndary, and deficiencies in second- (and first-) chance systems
Outpouring of promising innovations addressing these issues
But these policies and systems remain small scale and highly fragmented…
Enter Career Pathways – a body of ideas and practices for integrating
promising innovations promoted by a growing movement in the U.S. And,
with work, a conceptual framework for building systematic, high-quality
evidence on these ideas and practices
– It’s not about work vs. education, it’s about joining the two
– I.e., “We’ve got the… stuff, we’ve just got to bring it together!” (Obama, on the
economy in general, 9-28-12)
4
Building Evidence on Career
Pathways that Counts
 Abt Associates, a global social policy research and technical
assistance firm, is leading several national random assignment
studies of promising career pathways interventions
 The work has involved mapping out a conceptual framework and
identifying priorities and opportunities for useful experiments
 I will introduce this framework, show how it can help to organize a
wide range of studies, and discuss some ways we are designing
these projects so that their findings will count
5
Key Career Pathways Ideas
 Address the wide range of skills and needs of second-chance
populations
 Create manageable, well-articulated training steps
 Provide credentials valued in high demand occupations/sectors
 Build effective partnerships
– E.g., public agencies (education, labor, human services); community colleges;
community-based organizations; employers
– Vs. K-12, more diverse actors, complexity
 Basic ideas also being applied in strengthening career pathways in
secondary education
6
Prospects for good-paying, stable employment
The Basic Career Pathways Model
V. BA+ Programs
Upper-Skilled Jobs
IV. 1-2-Year Certificate to AA Programs
Mid-Level Skilled Jobs
III. Short-Term Certificate Programs
Entry-Level Skilled Jobs
II. Sectoral Bridge Programs
Skilled Jobs
Semi-
I. Basic Bridge Programs
Occupational, academic, and life skills
Program Inputs:
Signature CP Service Strategies
 Comprehensive assessment
– Academic and non-academic skills
 Basic and technical skills instruction
– Modularization, contextualization, acceleration, flexible delivery,
active learning
 Supports
– Pro-active advising and guidance, supplemental instruction,
social supports, supportive services, financial assistance
 Employment connections
– During and after training
8
Take First/ Next
Step In Career
Pathway/Lattice
Participant Characteristics
Theory of Change for
Career Pathways
• Demographic
• Educational
• Economic
Increase Performance &
Persistence in Training
Foundational Academic Skills
• Certificate/Diploma
• 2- year, 4- year Degree
Occupational Skills
Comprehensive
Assessment
Psycho-social Factors
Improve Performance &
Advancement in Jobs
Core Curriculum
Career Orientation and Knowledge
Supports
•  Earnings
•  Benefits
•  Job security
Resource Constraints
Employment
Connections
Improve Other
Outcomes
Other Personal and Family Challenges
• Income & assets
• Child & adult well being
• Local economic growth
Contextual Factors: Institutional, Economic, Social
Program Inputs
Intermediate Outcomes
Primary Outcomes
TO NEXT STEP
Initial
Targeting &
Placement
Decisions
Illustration #1: Pima Community
College Pathways to Healthcare
Program (Arizona)
• Basic elements
– In partnership with county One-Stop, college provides training and
credentials in 16 health care occupations, organized into 5 broad
pathways (+ 3 broad levels), programs varying from 1-24 months.
Targets low-income adults, substantially Latino population. Funded by
federal HPOG program.
 Signature strategies
– Instruction: Upfront assessment and training planning, w/10-week
contextualized basic skills development program if below required entry
skill levels. Many programs offer streamlined/compressed formats,
translate “clock-hours” at lower levels to prerequisites to higher levels.
– Supports: More intensive and coordinated case management (via OneStop) and academic advising and services (via college), financial
support covers training and related expenses.
– Employment: Outreach to engage local employers in training,
internships, employment; use One-Stop services; MOUs linking to
various county and state agencies.
10
Illustration #2: Year Up (National)
• Basic elements
– National non-profit operating in 8 urban areas. Partners with community
colleges and major employers in financial services and information
technology. Targets disadvantaged youth, 18-24, with high school
diploma/equivalent.
 Signature strategies
– Instruction: Six-month “learning development” phase provides
customized training in technical and contextualized basic and
professional skills (e.g., communication, behavior). Earn 14+ college
credits through agreement w/local colleges.
– Supports: “High support, high feedback” environment—learning
communities, pro-active counseling, structured group sessions,
supportive services, and weekly performance-based stipend of up to
$260 throughout the year. Post-program follow-up and engagement.
– Employment: Second six-month internship phase places students in
entry-level financial services/IT positions with major employers, who
contribute about half of total program costs per participant.
11
Illustrative Evidence on More Explicit Career
Pathways Approaches: Comprehensive, well-targeted
“first step” training programs can have substantial
impacts
Sectoral training (Maguire et al. 2010)
•
3 experienced CBOs provide customized
short-term training in varied high-demand
fields
•
Disadvantaged adults, HS+, careful
screening (n=1,014)
Year Up (Roder & Elliott 2011)
•
National org provides 6 months
customized training + 6 month paid
internship in IT and finance sectors
•
Disadvantaged urban youth 18-24, HS+
•
Program articulates w/local colleges,
completers earn 14+ credits
•
Results from small initial experiment
(n=164)
12
Examples of Recent Experimental
Findings on Signature CP Strategies
 Learning communities
– As bridge program (Fein et al. 2006)
– By linking courses (Sommo et al. 2012; Visher et al. 2012)
 More intensive, specialized personal guidance and support
– Coaching (Bettinger & Baker 2011)
– Help with financial aid applications (Bettinger et al. 2012)
– Student success courses + support (Weiss et al. 2011)
 Increased financial support
– Performance-based scholarships (Patel & Richburg-Hayes 2012)
– Unconditional needs-based grants (Goldrick-Rab et al. 2011)
 Psycho-social interventions
– E.g., implicit theories of intelligence, normalizing worries about fitting in and
success (Yeager & Walton 2011)
13
Designing Evaluations that Matter
Evaluations should be designed in ways that invite practitioners to make use of the results and adopt
solid practices based on evidence. We need to recognize the role of motivated program leaders at the
center of evaluation efforts, to ensure that these efforts advance program theory and practice, rather
than merely fulfilling a funder request. Research tells us that organizations and their leaders need to
own and trust information in order to use it. Partnership on the ground between practitioners and
evaluators, along with the long-term support of committed public and private funders, is indispensable
if the goal is to deliver evaluations that actually improve program quality and effectiveness.
–
Public/Private Ventures, Priorities for a New Decade: Making (More) Social Programs Work (Better), 2011.
Principles





Elucidate and address theories of program effectiveness
Develop strong common metrics that will be easily understood and widely
accepted
Design to anticipate interest in scaling and replicating successful
approaches
Foster appreciation for technical rigor (e.g., experimental designs)
Produce evaluation products useful to policymakers and practitioners
– Emphasis on active interaction: evaluators-practitioner, practitioner-practitioner
14
Example: The ISIS Evaluation
 First national random assignment study of U.S. career pathways
programs
 Separate tests of nine relatively comprehensive career pathways
programs, with implementation and cost-benefit studies

Central goals
– Assess overall effectiveness of each program
– Understand “why and under what conditions?”
• Work within well-specified theory of change
• Strong focus on impacts on exposure to program inputs, intervening outcomes,
heterogeneity
• Non-experimental analysis of mediators of impacts on primary outcomes
 Strong emphasis on collaboration helps ensure findings will count…
15
Collaboration Increases Chances that
Findings Will Matter


Extensive stakeholder outreach + literature review identify career pathways
focus in ISIS…
Early and ongoing exchanges foster interest and create positive climate for
experiments among potential sites
–
–

Maintain and extend relationships with wider stakeholder network
–
–
–

Working w/sites as partners w/complementary expertise and interests
Building an ongoing “learning community”
• Learn from experts on the team, peer-to-peer exchange, participation in external forums
• Help plan and execute dissemination strategies
Including private and public funding agencies, policy makers, professional associates,
advocates, practitioners,
Through meetings, website, public webinars, varied publications
On diverse interests: related projects, substantive topics/directions in ISIS, evaluation
findings, evaluation products,
So that findings aren’t limited to a small number of reports, but shared in
dynamic, diversified ways with engaged audiences
16
Summary
 Career pathways framework useful in program and evaluation
design, organizing emerging findings
 Genuine partnerships with practitioners and other stakeholders key
to designing evaluations that matter
 Thank you!
17