Transcript California Afterschool Outcome Measures Project
afterschooloutcomes.org
Deborah Lowe Vandell, PhD.
Principal Investigator Professor and Founding Dean School of Education University of California, Irvine
Project Researchers:
Valerie Hall, Ph.D.
Pilar O ’Cadiz, Ph.D.
Andrea Karsh, M.A.
Ba c k gr oun d a nd Ra t i ona l e for Onlin e T
o o l b o
Research indicates high quality afterschool programs can have beneficial effects on students both socially and academically
x
The Afterschool Outcome Measures Project at UC Irvine developed an Online Toolbox in response to the State ’ s legislative mandate for alternative accountability measures to standardized test scores for publically funded afterschool programs in California* Online Toolbox includes reliable and valid measures of Positive Behavior Change and Skill Development for use by diverse afterschool programs Field Test of online toolbox at over 200 afterschool and summer learning program sites Online Toolbox Statewide Implementation in Fall 2012/ Spring 2013
Development and Testing of Outcome Measures
THREE PHASES
PHASE 1 —INSTRUMENT DEVELOPMENT (2008-2009) Literature review Survey of the field (expected outcomes and current practices) Selection of measures with input from the CDE and Accountability and Outcomes Subcommittee (representation from diverse programs) PHASE 2 —PILOT STUDY (2009-2010) Pre-piloting at 9 sites to test procedures and instructions & to determine feasibility of online administration of surveys • • Piloting at 28 sites to test protocols under varying conditions:
Range of program size, geographic location, and level of operation Sites with/without internet access
PHASE 3 —FIELD STUDY (2010-2011) • Test of measures & online administration across the State of California 196 field test sites representing diverse program sizes and populations served by 29 programs in 9 regions statewide
NEXT PHASE:
Statewide Implementation of Online Toolbox for California ASES & 21stCCLC Programs
Opportunity to participate open to all publically funded afterschool programs in California: Afterschool Education and Safety Program (ASES) 21 st Century Community Learning Center (21 st CCLC) Up to 1000 sites statewide in 2012-2013 Up to 2000 sites to participate by 2014 Representation of diverse programs across 11 regions in the state
WHAT MEASURES ARE IN TOOLBOX ?
WHO TAKES THE SURVEYS?
HOW DO PROGRAMS ADMINISTER SURVEYS?
WHEN DO ASSESSMENTS TAKE PLACE?
WHY DO IT?
Web-based surveys of student performance collected from three respondents: 1.
Students (grades 3-12) 2.
3.
Program staff Classroom teachers
PRE and POST Surveys collected at TWO time points, for example:
Time 1 (early Fall/Winter) & Time 2 (Spring)
Beginning of Summer & End of Summer
Technical Assistance provided through:
Online Instructions and Resources Email and Telephone Communication
Individualized reports are provided to all participating programs.
These reports enable programs to compare their students’ skills and behaviors over time, and to compare their program and site-level outcome scores to aggregate scores across the State.
Programs have option to administer one, two, or three surveys: (1) Student (grades 3-12); (2) Program Staff; (3) Classroom Teachers STUDENT SELF-REPORTS
• • • •
Skill Development
Work Habits Reading Efficacy Math Efficacy Science Efficacy • •
Positive Behavior Change
Social Competencies Reductions in Misconduct
PROGRAM STAFF & CLASSROOM TEACHER REPORTS
—of Student Performance Skill Development
• Work Habits • Task Persistence • Academic Performance (classroom teachers only) • • •
Program Quality
(post only) Affiliation with Peers Relations with Program Staff Interest in Program Activities
Positive Behavior Change
• Reductions in Aggressive Behaviors • Prosocial Behavior with Peers • Social Skills
Field Test Key Findings
The Online Toolbox is an easy-to-use, inexpensive method of assessing skill development and positive behavior.
Elementary and middle school students reported having high quality experiences at their ASES programs.
Quality of experiences at ASES programs is linked to students ’ skill development and positive behaviors.
Significant improvements in student performance were found between the fall and spring assessments.
SIGNIFICANT FINDINGS 2010-2011 Field Test
PRE-POST OUTCOMES in MATCHED SAMPLE
Staff Reports of Elementary Students Increase in Prosocial Behavior ** Staff Reports of Middle School Students Increase in Prosocial Behavior ** Teacher Reports of Elementary Students Increase in Academic Performance ** Decrease in Aggressive Behavior** Student Reports of Quality Program Experience Positive student reports of program experience linked to positive results in ALL student outcomes ** p<.01
FINDINGS
Staff Reports on Student Prosocial Behavior
Increase in pre-post scores —matched sample
1,43 1,42 1,41 1,4 1,39 1,38 1,37 1,36 1,35 1,34
Elementary School Students 1,37 1,42
Fall Spring 1,4 1,38 1,36 1,34 1,32 1,3 1,28 1,26 1,24 1,22
Middle School Students 1,39 1,28
Fall Spring
Teacher Reports on Elementary Students
3,2 3,1 3 2,9 2,8 2,7 2,6
2,79
Fall
3,13
Spring 0,4 0,35 0,3 0,25 0,2 0,15 0,1
0,35
Fall Spring
0,28
Program Quality Distribution
Staff & Activities 45,0% 40,0% 35,0%
39,3% 34,3% 31,9% 34,3%
30,0% 25,0% 20,0%
25,9%
15,0% 10,0% 5,0%
5,5% 9,9%
0,0% Low Fair Elementary Good Middle
18,9%
Excellent Peer Affiliation 45,0% 40,0% 35,0%
32,6% 32,8% 30,1%
30,0% 25,0% 20,0% 15,0%
24,8%
10,0% 5,0% 0,0%
5,4% 7,1%
Low Fair Elementary Good Middle
39,7% 27,4%
Excellent
Quality to Outcomes
3,8 Student reports of higher quality experiences with staff & activities are linked to greater Math Efficacy 0,70 3,63 3,6 0,60 3,49 3,47 3,4 3,20 0,50 3,2 3,13 Student reports of higher quality experiences with peers are linked to less Aggressive Behavior 0,60 0,64 0,45 0,53 0,43 3 0,40 2,8 2,73 0,34 0,30 2,6 2,4 0,20 2,2 2 0,10 Low Quality Mid Quality Elementary Middle High Quality Low Quality Mid Quality Elementary Middle High Quality
EASY TO ADMINISTER ONLINE SURVEYS
Step-by-Step Instructions available to download from website http:// afterschooloutcomes.org
/
Sign-up & Instructions on Website
To APPLY to participate in Statewide Implementation CLICK HERE
To download Instructions for Administering Surveys
CLICK HERE
Program Staff & Teacher Surveys
Reports on Students in Afterschool Program Program Staff and Classroom teachers are assigned individual access codes Use both individual access code and the ID code for each student for whom they complete a report Each
student report
takes
3-5 minutes
to complete.
Student reports may be completed in multiple sessions. For example: A staff completing reports for 20 students can do four reports a day (a 20-minute session) on five different days.
Provides a battery of valid, reliable, and easy to administer youth outcome measures Expands accountability tools beyond standardized test scores and other academic measures • Value in tracking individual student performance over time with matched pre-post surveys, providing more powerful & interpretable data MATCHED = Respondent data are matched for both pre and post survey administrations (i.e. same students, staff and/or teachers in fall and spring) Information can be useful to guide program improvement based on student reports of program experience AND on student performance
For more information on measures:
Forum for Youth Investment publication
(September 2011) • Review of measures in Toolbox with a technical appendix •
Down Load at:
forumfyi.org
Or, link from our website:
http://afterschooloutcomes.org/