Transcript Document
What We Have Learned?
Research & Evaluation Presentation
Charles Smith, Stephen Baker, Neil Naftzger
Leadership Institute
Thursday, May 9
SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
• What does the research say about how
quality afterschool programming makes a
difference for youth?
• Why should we invest in evaluating the
quality of afterschool programs?
SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Four Continuous Improvement Practices4
Organization Level Setting
(Plus 10 hours of TA/coaching for site managers to implement the four CI practices)
SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Theory of Change Template for Afterschool
Outcomes8
SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Point of Service
Setting
l
Organization Setting
Policy
Setting
2012-2013 Dissemination
85 Networks/ Systems
>3250 Sites
>21,125 Staff
Estimate based on mean of 6.5
staff per site in YPQI Study Sample
>276,250 Child &
Youth
Estimate based on mean daily
attendance of 85 youth per day in YPQI
Study Sample
SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
System Accountabilities: Higher Stakes
Higher Stakes
Accountabilities
Objective
Data
Publicity
Action
Improved
Outcomes
SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
System Accountabilities: Lower Stakes
Higher Stakes
Accountabilities
Objective
Data
Meaningful
Information
Interpretive
Community
Action/
Expertise
Improved
Outcomes
Team Planning and
Implementing
•Team Self Assessment
•Improvement planning
•Review external scores
•Performance coaching
Lower Stakes Accountabilities
SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
System Supports for CI Practices
Policy:
Eligibility, Targeting,
Low/high stakes
Training, TA
& Coaching
Evaluation
External Raters, Program
Evaluation
SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Program Objectives
N=100 site
managers
Objective 1 (%)
Objective 2 (%)
Objective 3 (%)
Academic
Support
56
63
27
Social/civic
Artistic
development development
29
18
32
2
3
10
Sports &
Recreation
6
11
27
Other
(computer skills,
self-esteem, parent
workshop, etc.)
7
4
5
SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Content Offerings at Baseline
Leadership
Reading
Life Skills
Art
Physical Fitness
Technology/Computers
Math
Community Service
Sports
Creative Writing
Cooking
Science
Dance
Music
Theater
Poetry
Building/Shop
Percentage of sites
(N=87 site
managers)
97
96
95
93
91
90
89
89
86
78
77
76
71
71
69
49
35
Example program offerings
Planning team for events, youth advisory board
Vowels, spelling
Discuss race and culture tension
Scrapbooking, clay
Walleyball, gym
Typing & navigating skills, video production
Ratios, counts re: food drive donations
Christmas gifts to those in shelters
Basketball, baseball
Journaling
Recipes and cookie dip
Laws of motion, inertia experiments
Hip Hop class
History of Pop Music, Guitar lessons
Rehearsed 3 acts for musical play Annie
Forms of poetry lesson
Lego robotics, building a tower out of paper
SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
Conclusions from Qualistar Evaluation
Third, the study examined the relationships between the Q-QRIS star ratings,
the individual Q-QRIS component measures that yield those ratings, and two
commonly used measures of adult-child interaction. (Adult-child interaction is
generally agreed to be the most important aspect of childcare quality.) The
study found that the star ratings and the Q-QRIS components are generally
unrelated to measures of staff -child interaction, but the lack of results may
reflect the fact that these latter quality measures were collected in only one
classroom per provider.
Fourth, according to the logic model underlying QRISs, an improved child-care
environment, characterized by more responsive caregiving and enriched
content, will lead to better outcomes for children. The researchers examined
relationships between the star ratings, Q-QRIS components, and child cognitive
and social outcome measures. The researchers found few relationships
between individual Q-QRIS components and child outcomes and virtually
none between star ratings and child outcomes.
SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?
What We Have Learned?
Research & Evaluation Presentation
Charles Smith, [email protected]
Leadership Institute
Thursday, May 9
SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?