Transcript THE AMERICAN DREAM - Forum for Youth Investment

Program Quality & Youth Outcomes
A Ready by 21®
Quality Counts
Webinar
March 13, 2008
The Forum for Youth Investment
© The Forum for Youth Investment 2008. Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment.
Submit a Question at Any Time!
• To ask a question, go to the “Question
and Answer” box in the upper righthand corner of your screen. Click on
the small arrow next to “Question and
Answer,” type in your question, and
click send.
© The Forum for Youth Investment 2008. Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment.
Agenda
•
•
•
•
•
•
•
Welcome, Review Goals, Speakers, Logistics - Nicole Yohalem
Making the Case for Quality - Karen Pittman
Population Level Outcomes & Indicators - Karen Finn
Program/System Level Quality & Outcomes Charles Smith
Summary Reflections - Karen Pittman
Question and Answer Period
Closing, Review Key Resources – Nicole Yohalem
© The Forum for Youth Investment 2008. Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment.
Why is Program Quality the
Focus of this Initiative?
Making the Case for Quality
Karen Pittman
The Forum for Youth Investment
© The Forum for Youth Investment 2008. Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment.
Improving Community Supports is the intermediate step
toward improving Youth Outcomes.
While improving youth outcomes is the goal we are all working
toward, we believe that to get there at scale, our focus needs to
be on helping leaders improve the quality and reach of
programs and services available in the community.
© The Forum for Youth Investment 2008. Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment.
Increasing program availability isn’t enough.
•
There is basic agreement that young people need programs and
services – in particular that they need more structured, voluntary
activities to fill their out-of-school hours.
•
There is a basic belief, and evidence, that participation in these
programs is associated with improved youth outcomes – academic,
social, emotional, civic, health, management of risky behaviors.
•
There is, consequently, growing public and political will, to increase
youth participation in programs and to ensure that the programs are of
good quality.
•
There is a growing base of research that suggests that improvement in
youth outcomes is strongly related to level of participation and level of
program quality – more refined measures of both are needed.
•
Measuring quality at the system level has always been challenging.
© The Forum for Youth Investment 2008. Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment.
What the research says.
•
“Dosage and duration” matter. Young people who participate regularly in
programs over a sustained period of time show gains in targeted
outcomes. Those who attend sporadically don’t.
•
Program quality also matters. Controlling for participation, young
people who participate in high quality programs achieve greater gains
than those who do not.
•
Recent meta-analysis by Durlak and Weissberg suggests that lowquality programs may have no effect on adolescent outcomes.
© The Forum for Youth Investment 2008. Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment.
Durlak and Weissberg study
•
Recent meta-analysis by Durlak and Weissberg suggests that low-quality
programs may have no effect on adolescent outcomes.
• 73 after-school programs reviewed grouped into 2 clusters based on
SAFE criteria:
• Sequenced set of activities to achieve goals
• Active learning techniques for skill acquisition
• Focus, to some extent, on personal/social development
• Explicit objectives for personal and/or social skill development
• Programs that had the SAFE features showed positive effects on
almost every outcome – school performance, social behavior, attitudes
and beliefs.
• Programs that did not have the SAFE features showed no effect on
any outcome.
© The Forum for Youth Investment 2008. Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment.
Implications for Programs, Public and
Policy Makers
•
Quality counts as much as participation.
• Two youth, spending equal time in two programs of different quality
will reap significantly different benefits.
• Overtime, the cumulative effect of participation in low quality
programs, especially when coupled with participation in low quality
schools and harried families, can be huge.
•
Improving program quality at the point of service – where youth and
adults interact – has the biggest impact on youth outcomes.
•
But program quality is either not measured, not measured well or not
measured consistently across programs.
•
This means that increasing funds for program expansion and
participation, without increasing funding and monitoring for quality, may
net mediocre results.
© The Forum for Youth Investment 2008. Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment.
POS Quality Meets
System Performance Measurement
• Results-based accountability helps program/system
directors answer 3 questions: How much do we do? How
well do we do it? Is anyone better off?
• These 3 questions can be easily linked to the concepts of
participation, POS quality, and youth outcomes.
Program/System Level Performance
PARTICIPATION
QUALITY
PROGRAM OUTCOMES
How much do we do?
How well do we do it?
Is anyone better off?
Frequency,
duration,
intensity of
participation
Point of
Service
Quality
Improved skills,
knowledge &
behaviors
As measured by indicators e.g., %
with improved decision-making skills
© The Forum for Youth Investment 2008. Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment.
Defining Terms
POPULATION LEVEL
Outcomes or Results: A condition of well-being for children, adults, families or
communities (e.g. youth ready for college, work, life).
Indicators: Measures that help quantify achievement of a result or outcome (e.g. rate
of high school graduation, teen birth rates, employment rates).
PROGRAM/SYSTEM LEVEL
Performance Measurement: Measuring how well a program, organization or system
is working:
•
Participation. Measuring frequency, duration and intensity of program
attendance.
•
Program Outcomes: Impacts or changes for participants during or after a
program, often expressed in terms of knowledge, skills or behaviors.
•
Program Quality: Youth access to developmentally significant experiences in a
setting. Although program quality can be defined more broadly, point of service
quality refers primarily to staff practices during specific activities with children
and youth.
Based on Mark Friedman’s Results-Based Accountability
© The Forum for Youth Investment 2008. Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment.
Defining Terms
Point of Service Quality:
Youth access to developmentally significant experiences in a
setting. Although program quality can be defined more broadly,
point of service quality refers primarily to staff practices during
specific activities with children and youth.
© The Forum for Youth Investment 2008. Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment.
P
O
P
U
L
A
T
I
O
N
L
E
V
E
L
G
O
A
L
S
FAMILY & COMMUNITY
CHILD & YOUTH
SUPPORTS or INPUTS
How is our community doing?
e.g.
OUTCOMES or RESULTS
How are our young people doing?
Caring Adults
e.g.
Safe Places
Effective Ed
Healthy Starts
Opportunities to
Contribute
Connecting
Thriving
Leading
Working
Learning
As measured by indicators
e.g., Teen birth rate
(common resources: Kids
Count; Search internal assets)
As measured by indicators e.g.,
% with caring adult
(common resources: Search
external assets)
Program/System Level Performance
PARTICIPATION
QUALITY
PROGRAM OUTCOMES
How much do we do?
How well do we do it?
Is anyone better off?
Frequency,
duration, intensity
of participation
Point of
Service
Quality
Improved skills,
knowledge & behaviors
As measured by indicators e.g., %
with improved decision-making skills
The Maryland Ready by 21
Action Agenda
Karen
Finn
The Forum for Youth Investment
Maryland’s Child Well-Being Results
• Babies born healthy
• Healthy Children
• Children enter school ready to learn
• Children are successful in school
• Children complete school
• Children are safe in their families and communities
• Stable and economically self-sufficient families
• Communities that support family life
© The Forum for Youth Investment 2008. Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment.
Youth Told Us…..
“They need to craft programs around older youths’
interests….bring in adults in the same field of interest and
partner them with youth…”
“….foster parents and youth workers…..show them how to
connect with kids….”
© The Forum for Youth Investment 2008. Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment.
Maryland’s Ready by 21 Action Agenda
P
O
P
U
L
A
T
I
O
N
L
E
V
E
L
G
O
A
L
S
ACTION AREAS
How is our community doing?
DESIRED RESULTS
How are our young people doing?
Competent, Caring Adults
Accessible, Affordable Housing
Access to Health Care
Pathways to Education and
Employment
Equal Treatment Under the Law
Ready for
College, Work
& Life
As measured by indicators
e.g., educational attainment,
employment, health care
coverage, HIV/AIDS rate…
Program/System Level Performance
PARTICIPATION
QUALITY
PROGRAM OUTCOMES
How much do we do?
How well do we do it?
Is anyone better off?
Frequency,
duration,
intensity of
participation
Point of
Service
Quality
Improved skills,
knowledge &
behaviors
Quality & Outcomes at the
Program/ System Level
Charles Smith
The Forum for Youth Investment
POS Quality and Program Outcomes:
OST Settings
• Durlak & Weissberg, 2007
• SAFE
• NRC Blue Book, 2002
• “research specifically linking…[setting] features to outcomes
is rare…”, that “we are only beginning to understand how the
different combinations of features in organized activities
interact to promote positive development…”, and finally that
“precisely which features are involved and how they co-act to
produce specific developmental change has not yet been
evaluated” (Mahoney et al., 2005, p. 12-13).
• Robert Halpern
© The Forum for Youth Investment 2008. Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment.
POS Quality and Program Outcomes:
Classroom Settings
• Research on Motivation:
• Relatedness, autonomy, and competence
• Research on Learning:
• Positive affect, active learning and meta-cognition
© The Forum for Youth Investment 2008. Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment.
What is actually happening in 599 OST settings?
Does prior research make sense to you?
welcome
5
belong
Quality
4
learning
group
3
choices
2
planning
reflect
1
PYD I
Engagement
Interaction
Supportive
Environment
Reflect
Plan
Choose
Encouragement
Skill building
Active engagement
Be in small groups
Staff Cent I
Low Qual II
5 = occurred for everyone
3 = occurred for some
1 = did not occur
Reframing conflict
Experience belonging
Welcoming atmosphere
Safe Environment
© The Forum for Youth Investment 2008. Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment.
Findings from Several Samples
• POS quality findings:
• Supportive environment related to: Attendance
• Interaction related to: Interest in program
• Engagement related to: Sense of challenge, sense of
growth, school-day reading, school-day suspension
• Note: No offerings get to high engagement without high
support and high interaction
• Quality Improvement (YPQI) Findings
© The Forum for Youth Investment 2008. Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment.
Are the Youth PQA and YPQI Improvement Model
RESEARCH BASED?
•
YES – The Youth PQA has been the subject of lots of research can be
considered a reliable and valid measurement instrument by widely
shared standards of evaluation
•
YES, conditional – The YPQI sequence has been tested in at least one
high fidelity implementation and use of the YPQI sequence is related to
POS quality change
•
IN PROCESS – The YPQI sequence is currently the subject of a
randomized field trial, results available in fall 2008
•
Fidelity to broad developmentally focused intervention models
© The Forum for Youth Investment 2008. Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment.
How to Talk about Outcomes
•
OST programs can produced positive change in social-emotional skills
and academic achievement for both at-risk and lower-risk populations of
children and youth.
•
Not all OST programs produce positive change so we should be careful
to protect investment by doing quality assurance. It’s cost effective to
ensure that staff are actually doing good stuff.
•
Programs can raise child outcomes by striving to improve the quality of
their offerings.
•
Broad developmentally-focused programming is a research-based
strategy.
© The Forum for Youth Investment 2008. Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment.
Question & Answer Period
• To ask a question, go to the “Question
and Answer” box in the upper righthand corner of your screen. Click on
the small arrow next to “Question and
Answer,” type in your question, and
click send.
© The Forum for Youth Investment 2008. Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment.
Program Quality & Youth
Outcomes
A Ready by 21 Quality
Counts Webinar
March 13, 2008
The Forum for Youth Investment
© The Forum for Youth Investment 2008. Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment.
Key Readings
Population Level
•
•
•
•
•
Trying Hard is Not Good Enough (Mark Freidman)
www.resultsaccountability.com
Finding Out What Matters for Youth (Gambone, Klem & Connell, 2002)
www.ydsi.org/ydsi/pdf/WhatMatters.pdf
Maryland’s Ready by 21 Five-Year Action Agenda
www.ocyf.state.md.us/
Forum Focus: What Gets Measured Gets Done
www.forumfyi.org/node/149
Steering a Course Toward Effective Youth Policies: Dashboards for Youth
www.forumfyi.org/node/68
Program/System Level
•
•
•
•
•
YPQA validation study (Smith & Hohmann, 2005)
www.highscope.org/Content.asp?ContentId=308
Afterschool Quality and School-Day Outcomes (Blazevski & Smith, 2007)
www.highscope.org/Content.asp?ContentId=308
Improving Afterschool Program Quality (Granger, Durlak, Yohalem, Reisner, 2007)
www.wtgrantfoundation.org/usr_doc/Improving_After-School_Program_Quality.pdf
The Impact of After-school Programs…(Durlak & Weissberg, 2007)
www.casel.org/downloads/ASP-Full.pdf
Participation During Out-of-School Time (Forum for Youth Investment, 2004)
www.forumfyi.org/node/80
© The Forum for Youth Investment 2008. Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment.
Program Quality &
Youth Outcomes
A Ready by 21
Quality Counts
Webinar
March 13, 2008
The Forum for Youth Investment
© The Forum for Youth Investment 2008. Ready by 21 and the Ready by 21 Logo are registered trademarks of the Forum for Youth Investment.