Transcript Document
What We Have Learned? Research & Evaluation Presentation Charles Smith, Stephen Baker, Neil Naftzger Leadership Institute Thursday, May 9 SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE? • What does the research say about how quality afterschool programming makes a difference for youth? • Why should we invest in evaluating the quality of afterschool programs? SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE? Four Continuous Improvement Practices4 Organization Level Setting (Plus 10 hours of TA/coaching for site managers to implement the four CI practices) SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE? Theory of Change Template for Afterschool Outcomes8 SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE? Point of Service Setting l Organization Setting Policy Setting 2012-2013 Dissemination 85 Networks/ Systems >3250 Sites >21,125 Staff Estimate based on mean of 6.5 staff per site in YPQI Study Sample >276,250 Child & Youth Estimate based on mean daily attendance of 85 youth per day in YPQI Study Sample SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE? System Accountabilities: Higher Stakes Higher Stakes Accountabilities Objective Data Publicity Action Improved Outcomes SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE? System Accountabilities: Lower Stakes Higher Stakes Accountabilities Objective Data Meaningful Information Interpretive Community Action/ Expertise Improved Outcomes Team Planning and Implementing •Team Self Assessment •Improvement planning •Review external scores •Performance coaching Lower Stakes Accountabilities SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE? System Supports for CI Practices Policy: Eligibility, Targeting, Low/high stakes Training, TA & Coaching Evaluation External Raters, Program Evaluation SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE? Program Objectives N=100 site managers Objective 1 (%) Objective 2 (%) Objective 3 (%) Academic Support 56 63 27 Social/civic Artistic development development 29 18 32 2 3 10 Sports & Recreation 6 11 27 Other (computer skills, self-esteem, parent workshop, etc.) 7 4 5 SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE? Content Offerings at Baseline Leadership Reading Life Skills Art Physical Fitness Technology/Computers Math Community Service Sports Creative Writing Cooking Science Dance Music Theater Poetry Building/Shop Percentage of sites (N=87 site managers) 97 96 95 93 91 90 89 89 86 78 77 76 71 71 69 49 35 Example program offerings Planning team for events, youth advisory board Vowels, spelling Discuss race and culture tension Scrapbooking, clay Walleyball, gym Typing & navigating skills, video production Ratios, counts re: food drive donations Christmas gifts to those in shelters Basketball, baseball Journaling Recipes and cookie dip Laws of motion, inertia experiments Hip Hop class History of Pop Music, Guitar lessons Rehearsed 3 acts for musical play Annie Forms of poetry lesson Lego robotics, building a tower out of paper SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE? Conclusions from Qualistar Evaluation Third, the study examined the relationships between the Q-QRIS star ratings, the individual Q-QRIS component measures that yield those ratings, and two commonly used measures of adult-child interaction. (Adult-child interaction is generally agreed to be the most important aspect of childcare quality.) The study found that the star ratings and the Q-QRIS components are generally unrelated to measures of staff -child interaction, but the lack of results may reflect the fact that these latter quality measures were collected in only one classroom per provider. Fourth, according to the logic model underlying QRISs, an improved child-care environment, characterized by more responsive caregiving and enriched content, will lead to better outcomes for children. The researchers examined relationships between the star ratings, Q-QRIS components, and child cognitive and social outcome measures. The researchers found few relationships between individual Q-QRIS components and child outcomes and virtually none between star ratings and child outcomes. SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE? What We Have Learned? Research & Evaluation Presentation Charles Smith, [email protected] Leadership Institute Thursday, May 9 SYSTEM-BUILDING AND QUALITY: WHAT’S AT STAKE?