Transcript GOOD MORNING EVERYONE - Forum for Youth Investment

ASSESSING AND IMPROVING
YOUTH PROGRAM QUALITY
Nicole Yohalem
October 2008
Outline
• Intro discussion
• Making the case for investing in quality
• Where is the field?
• Defining Quality
• Assessing Quality
• Improving Quality
© The
Forum for Youth Investment 2008
Features of Positive Developmental Settings
National Research Council, 2002
• Physical and Psychological Safety
• Appropriate Structure
• Supportive Relationships
• Opportunities to Belong
• Positive Social Norms
• Support for Efficacy and Mattering
• Opportunities for Skill-Building
• Integration of Family, School and Community efforts
© The
Forum for Youth Investment 2008
National Research Council
Recommendation
“Communities should put in place some locally appropriate
mechanism for monitoring the availability, accessibility and
quality of programs…”
- Community Programs to Promote Youth Development, 2002
© The
Forum for Youth Investment 2008
Discussion
• In your organizational context, which of
these features do you consider strengths?
• Which represent areas of growth?
• Are you trying to measure any of these in
your settings?
• Any common themes at your table?
© The
Forum for Youth Investment 2008
Making the Case: Why focus on quality?
1. Because too few young people get the supports
they need to thrive.
•
According to the America’s Promise Alliance National Promises
Survey, only 31% of 6-17 year olds have at least 4 of the 5
promises. 21% have 1 or none.
1.
2.
3.
4.
5.
© The
Forum for Youth Investment 2008
Safe Places
Caring Adults
Effective Education
Opportunities to Help Others
Healthy Start
Why focus on quality?
2. Because participating in OST programs can
make a difference.
•
Several empirical reviews of the effects of programs conducted
over the past decade show that on average, programs have
positive effects on social, emotional and academic outcomes.
(Durlak & Weissberg, 2007; Bodilly & Beckett, 2005; Lauer et al., 2006).
© The
Forum for Youth Investment 2008
Why focus on quality?
3. Because the quality of OST program matters.
Durlak and Weissberg, 2007
• 73 programs reviewed grouped into 2 clusters based on SAFE
criteria: (Sequenced, Active, Focused, Explicit)
• Programs that had the SAFE features showed positive effects on
almost every outcome – school performance, social behavior,
attitudes and beliefs.
• Programs that did not have the SAFE features showed no effect on
any outcome.
Other studies: Vandell, MARS, High/Scope’s YPQA Validation Study
© The
Forum for Youth Investment 2008
Why focus on quality?
4. Because lots of children and youth spend
time in these settings.
•Afterschool Alliance estimates 6.5 million children are in
after-school programs. The parents of another 15.3 million
say their child would participate were programs available.
•State of California moving toward universal access –
elementary and middle.
© The
Forum for Youth Investment 2008
Why focus on quality?
5. Because significant public and private
dollars flow into these settings.
• The Finance Project estimated a $3.6 billion federal investment in
after-school 2002.
• State and local funds increasingly important.
• California’s recent commitment of $550 million.
• Major private investments by national foundations (Wallace, Mott
Atlantic, Clark, Robert Wood Johnson, W.T. Grant) and regional
foundations (Nellie Mae, Skillman, William Penn, Colorado Trust).
© The
Forum for Youth Investment 2008
Why focus on quality?
6. Because we know quality varies significantly
within and across settings.
• All programs are not created equal (Durlak & Weissberg).
• Despite the variation, there are some patterns. Overall, programs do
better on traditionally regulated things like safety than they do on higherorder things like engagement, interactions.
• Not all OST programs produce positive change. Funders can protect
their investments by doing quality assurance.
© The
Forum for Youth Investment 2008
Why focus on quality?
7. Because quality is measurable, and even more
important – it is malleable.
Palm Beach County, FL. Programs conduct assessments, sites
develop improvement plans targeting specific areas, receive quality
coaching, gains made in every area targeted by staff.
American Camp Association. Based on youth surveys, leadership
sets initial targets in partnership with youth and staff, develop an
action plan, re-do survey. 17 out of the 23 camps had positive
change in at least one area.
© The
Forum for Youth Investment 2008
Why focus on quality?
8. Because measuring program quality is an
efficient, affordable, and productive alternative (or
complement) to measuring youth outcomes.
• There is more agreement in the field about the components of quality
than about appropriate outcomes.
• Demonstrating impact is important, but doing it in such a way that
individual programs can take credit for the change is extremely
difficult, expensive and not necessarily instructive.
© The
Forum for Youth Investment 2008
Why focus on quality?
1.
2.
3.
4.
5.
6.
7.
8.
© The
Too few young people are getting the supports they need to
thrive.
Participating in community programs can make a difference.
The quality of community programs matters.
Lots of children and youth spend time in these settings.
Significant public and private dollars flow into these settings.
We know quality varies significantly with and across settings.
Quality is measurable and malleable.
Measuring program quality can is an efficient, affordable,
and productive alternative (or complement) to measuring
youth outcomes.
Forum for Youth Investment 2008
Why focus on quality?
• Quality matters.
• Quality is measurable.
• Quality is malleable.
© The
Forum for Youth Investment 2008
Quality and Reach Count
Outcome Areas
21
.
Ages
?
.
.
0
?
At best, school only fills
a portion of developmental space
© The
Forum for Youth Investment 2008
(Cognitive, Physical, Social,
Civic, Vocational…)
?
School
Morning
After
School
...
Times of Day
Night
Where is the field?
• Attention to and interest in quality has increased
•
•
•
•
© The
significantly over the past several years among key
stakeholder groups in OST.
There is consensus about what matters.
Increasingly, evidence suggests our focus should be on
the point of service.
A lot of work is underway to refine assessment tools and
develop systemic approaches to quality improvement.
Washington is a site in the Ready by 21 Quality Counts
initiative.
Forum for Youth Investment 2008
Measuring Youth Program Quality:
A Guide to Assessment Tools
•
Assessing Afterschool Program Practices Tool (APT)
National Institute on Out-of-School Time and the MA Department of Education
•
Out-of-School Time Observation Instrument (OST)
Policy Studies Associates
•
Program Observation Tool (POT)
National Afterschool Association
•
Program Quality Observation (PQO)
Deborah Vandell and Kim Pierce
•
Promising Practices Rating Scale (PPRS)
WI Center for Education Research and Policy Studies Associates, Inc.
•
Quality Assurance System (QAS)
Foundations Inc.
•
Program Quality Self-Assessment Tool (QSA)
New York State Afterschool Network
•
School-Age Care Environment Rating Scale (SACERS)
Frank Porter Graham Child Development Center, UNC
•
Youth Program Quality Assessment (YPQA)
High/Scope Educational Research Foundation
© The
Forum for Youth Investment 2008
Measuring Youth Program Quality:
A Guide to Assessment Tools
• Purpose and History
• Content
• Structure and Methodology
• Technical Properties
• User Considerations
• Application in the Field
© The
Forum for Youth Investment 2008
How is quality defined?
• There is a lot of similarity across definitions.
Common elements include:
• Relationships
• Environment
• Engagement
• Social/Behavioral Norms
• Skill Building Opportunities
• Routine/Structure
© The
Forum for Youth Investment 2008
© The
Forum for Youth Investment 2008
How is quality measured?
• There are more differences in how quality is
measured than in how it is defined.
• Why the differences in emphasis and approach?
• Tool purposes (regulatory, self-assessment)
• Program purposes (achievement , recreation)
• Developers’ perspectives & backgrounds (ECE, YD)
© The
Forum for Youth Investment 2008
Differences in emphasis
Social Processes/
Interactions
Resources
(financial, human,
material)
© The
Forum for Youth Investment 2008
SACERS
YPQA
9 items
14 items
35 items
16 items
Differences in approach
• Data collection methods
• Types of measures
• Technical properties
© The
Forum for Youth Investment 2008
Data collection methods
• Type (observation, interview, questionnaire,
document review)
• Target users (line staff, program leaders, youth,
external observers)
• Intensity of data collection
© The
Forum for Youth Investment 2008
Types of measures: high vs. low inference
•
Program Observation Tool
•
•
Staff are engaged with children
Youth Program Quality Assessment
During activities, staff generally smile, use friendly gestures, and make eye
contact.
• Staff encourage all youth to try out new skills or attempt higher levels of
performance.
• During activities, staff are almost always actively involved with youth (e.g.
they provide directions, answer questions, work as partners or team
members, check in with individuals or groups).
• Staff make use of frequent open-ended questions.
•
© The
Forum for Youth Investment 2008
Types of measures: diagnostic vs. prescriptive
• Diagnostic – NY QSA
• A quality program provides participants with a variety of engagement
strategies.
• Diagnostic and prescriptive - APT
• Youth are busy and engaged in conversation or activities.
• Youth appear relaxed and in control of themselves.
• Youth independently gather resources, materials or get information.
• Youth help select, lead or contribute to the running of the activity.
• Youth solve problems alone or in groups.
• When trying to solve a problem, youth try to identify the source, nature of
the problem and/or try out potential solutions.
© The
Forum for Youth Investment 2008
Technical properties
© The
Forum for Youth Investment 2008
The YPQA: Assessing point of service quality
The YPQA is a good example of new class of tools that:
• Produce data that leads to real change in staff
performance
• Provide continuity that is place-based (not silo-based)
• Link accountability policy with workforce development
• Offer a more efficient and effective use of resources
than a sole focus on measuring child outcomes
© The
Forum for Youth Investment 2008
The YPQA: Assessing point of service quality
Plan
Make choices
Engagement
Reflect
Lead and mentor
Be in small groups
Partner with adults
Experience belonging
Interaction
Encouragement Reframing conflict
Supportive
Skill building
Session flow
Environment
Active engagement Welcoming atmosphere
Psychological and emotional safety
Program space and furniture
Emergency procedures
Safe
Healthy food and drinks
Physically safe environment
Environment
© The
Forum for Youth Investment 2008
Quality in context: Multi-level systems
Engage
Interaction
POS
Point Of Service
PLC
Professional
Learning Community
SAE
System Accountability
Environment
© The
Forum for Youth Investment 2008
Support
Safety
Youth PQA scores
5
4.38
4
3.72
2.97
3
2.53
2
1
I. Safe
Environment
© The
Forum for Youth Investment 2008
II. Supportive
Environment
III. Interaction
Opportunities
IV. Engaged
Learning
What quality looks like on the ground
welcome
5
belong
Quality
4
learning
group
3
choices
2
planning
reflect
1
PYD I
Staff Cent I
Low Qual II
“Positive Youth
Development”
28%
“Staff
Centered”
39%
“Low quality”
33%
• Sample of nearly 600 different youth workers
© The
Forum for Youth Investment 2008
Approaches to Improving Quality
Approaches differ along several dimensions:
•
•
•
•
•
•
•
•
© The
Nature of Agency Involvement (mandatory/voluntary)
Level of Accountability (high stakes/low stakes)
Reach (universal/targeted)
Source of Expertise (internal/external capacity)
Focus of Change (organizational issues/staff practice)
Staff Level Targeted (targets leadership/line staff)
Type of Data Collected (high/low inference; diagnostic/prescriptive)
Support Strategy (one-on-one/group support)
Forum for Youth Investment 2008
Common lessons
• Quality assessment can advance multiple goals
• Data about their own practice is a powerful
motivator for staff
• Common language helps pave the way for
change
• Important to couple standards and assessment
with tangible supports
Building Quality Improvement Systems, 2007
© The
Forum for Youth Investment 2008
Quality improvement systems using the YPQA
YPQA is part of state and local quality improvement efforts:
–Washington, New York, Iowa, Kentucky, Oklahoma (part of Ready by 21 Quality Counts)
–Statewide 21st Century: Michigan, Maine, Minnesota, Rhode Island, New Mexico
–Cities and Counties: Rochester, Syracuse, Detroit, Grand Rapids, Palm Beach, St. Louis,
Nashville, Austin, Georgetown Divide CA, Indianapolis, Columbus IN, Chicago
Rochester
Grand
Rapids
Minneapolis
Washington*
Chicago
New York
etroit
Iowa
Indianapolis
mbus
Georgetown
Divide
Columbus
St. Louis
Oklahoma
Rhode
` Island
`
Kentucky
Nashville
l
Austin
West Palm Beach
County
© The
Forum for Youth Investment 2008
`
Core components of a
quality improvement system
•
•
•
•
•
© The
Self-assessment
External assessment
Action planning (with data)
Training
Coaching/advising
Forum for Youth Investment 2008
Example: Palm Beach, 60 sites county-wide; multi-year model
Components
•
•
•
•
•
IMPROVEMENT
STEP 2a
Program Selfassessment
STEP 3
Plan for
improvement
STEP 4
Carry out
plan
Self-assessment
External assessment
Action planning with data
Training
Coaching
STEP 6
Re-assess
and move
forward!
STEP 1
Decide to
build system
STEP 2b
External
assessment
MONITORING
© The
Forum for Youth Investment 2008
STEP 5
External
assessment
w/ criterion
ACCOUNTABILITY
Example: Minnesota 115 sites statewide; multi-year model
Components
• Self-assessment
• External Data Collection
• Action Planning (with Data)
• Training
• Coaching
IMPROVEMENT
STEP 2a
Program Selfassessment
STEP 3
Plan for
improvement
STEP 4
Carry out
plan
STEP 6
Re-assess
and move
forward!
STEP 1
Decide to
build system
STEP 2b
External
assessment
MONITORING
© The
Forum for Youth Investment 2008
STEP 5
External
assessment
with criterion
ACCOUNTABILITY
What does the future hold?
A QIS will gain more traction and more likely be effective and
sustainable if the community/system also has solid information
about:
The overall program landscape (the full range of providers that
could be engaged in quality improvement)
• The youth-serving workforce (who youth workers are,
backgrounds and ambitions, turnover rates, full vs. part-time, etc.)
• Program participation
• Child and youth outcomes
•
•
© The
And if this information is part of an integrated information system
Forum for Youth Investment 2008
Measuring Youth Program Quality:
A Guide to Assessment Tools
www.forumfyi.org/node/297
Building Quality Improvement Systems
www.forumfyi.org/node/299
Nicole Yohalem
[email protected]