California Afterschool Outcome Measures Project
Download
Report
Transcript California Afterschool Outcome Measures Project
ONLINE TOOL BOX
UNIVERSITY OF CALIFORNIA, IRVINE
CAOMP Email: [email protected]
Background and Rationale for Online Toolbox
Research indicates high quality afterschool programs can have beneficial effects
on students both socially and academically
The California Outcome Measures Project (CAOMP) developed an
Online Toolbox in response to the State’s legislative mandate for alternative
accountability measures to standardized test scores for publically funded
afterschool programs*
Toolbox includes reliable and valid measures of Positive Behavior Change and
Skill Development to be used by State funded programs as part of their
accountability measures
Field Test of measures administered using online toolbox at over 200
afterschool and summer learning program sites concluded in Fall 2011
Online Toolbox will be made available for general use in Fall 2012
*In partnership with UC Davis and with support David and Lucile Packard Foundation &
California Department of Education
OVERVIEW
Afterschool Outcome Measures Online Toolbox
Web-based surveys of student performance collected from three
respondents:
1. Students (grades 3-8)
2. Program staff
3. Classroom teachers
PRE and POST Surveys collected at TWO time points, for example:
Early Fall & Late Spring
Beginning of Summer & End of Summer
Technical Assistance provided through:
Online Instructions and Resources
Email and Telephone Communication
Programs receive reports of their survey results at program and site
level and aggregate results of California statewide data for comparison
DESCRIPTION OF MEASURES
Programs have option to administer one, two or all three surveys:
(1) Student (grades 3-8; (2) Program Staff; and/or (3) Classroom Teachers
STUDENT SELF-REPORTS
—Elementary & Middle (5-6 subscales)
Skill Development
Work Habits
Reading Efficacy
Math Efficacy
Task Persistence (middle school only)
Positive Behavior Change
Social Competencies
Misconduct
Program Quality (spring only)
Staff & Activities
Peer Affiliation
PROGRAM STAFF &
CLASSROOM TEACHER REPORTS
—of Elementary & Middle School Student
Performance (5-6 subscales)
Skill Development
Work Habits
Task Persistence
Academic Performance (classroom teachers only)
Positive Behavior Change
Aggressive Behavior with Peers
Prosocial Behavior with Peers
Social Skills
SIGNIFICANT
FINDINGS
From 2010-11 Field Test
PRE-POST OUTCOMES in MATCHED SAMPLE
Staff Reports of Elementary Students
Increase in Prosocial Behavior **
Staff Reports of Middle School Students
Increase in Prosocial Behavior **
Teacher Reports of Elementary Students
Increase in Academic Performance **
Decrease in Aggressive Behavior**
Student Reports of Quality Program Experience
Positive student reports of program experience linked to positive
results in ALL student outcomes
** p<.01
* p<.05
SIGNIFICANT FINDINGS
Staff Reports on Student Prosocial Behavior
Increase in pre-post scores—matched sample
Elementary School Students
Middle School Students
1.4
1.43
1.39
1.42
1.42
1.38
1.41
1.36
1.4
1.34
1.39
1.32
1.38
1.3
1.37
1.28
1.37
1.36
1.26
1.35
1.24
1.34
1.28
1.22
Fall
Spring
Fall
Spring
SIGNIFICANT FINDINGS
Teacher Reports on Elementary Students
Academic Performance
Increase in pre-post scores—
matched sample
3.2
Aggressive Behavior
Decrease in pre-post scores—
matched sample
0.4
3.13
0.35
3.1
0.35
3
0.3
0.28
2.9
2.8
0.25
0.2
2.79
2.7
0.15
2.6
0.1
Fall
Spring
Fall
Spring
Program Quality Distribution
Peer Affiliation
Staff & Activities
45.0%
45.0%
39.3%
40.0%
34.3%
35.0%
39.7%
40.0%
34.3%
35.0%
32.6%
31.9%
30.0%
30.1%
30.0%
27.4%
25.9%
25.0%
18.9%
15.0%
10.0%
24.8%
25.0%
20.0%
32.8%
20.0%
15.0%
9.9%
10.0%
5.5%
5.0%
7.1%
5.4%
5.0%
0.0%
0.0%
Low
Fair
Elementary
Good
Middle
Excellent
Low
Fair
Elementary
Good
Middle
Excellent
Quality to Outcomes
Student reports of higher quality
experiences with staff & activities are
linked to greater Math Efficacy
3.8
Student reports of higher quality
experiences with peers are linked to
less Aggressive Behavior
0.70
3.63
3.6
3.49
0.64
3.47
0.60
0.60
3.4
3.2
0.53
3.20
3.13
0.50
0.45
0.43
3
0.40
2.8
2.73
0.34
2.6
0.30
2.4
0.20
2.2
2
0.10
Low Quality
Mid Quality
Elementary
Middle
High Quality
Low Quality
Mid Quality
Elementary
Middle
High Quality
How does it work?
EASY TO ADMINISTER
Step-by-Step Instructions Available to
download from
CAOMP Website
afterschooloutcomes .org
Updated Website
to Launch
JANUARY 2012
Online Toolkit
Available to public
FALL 2012
GO TO PROJECT WEBSITE:
afterschooloutcomes.org
1. Click on “Field Test Surveys” tab at top.
2. SELECT appropriate survey:
Once survey link is selected
you should see this first page
of the survey.
Example:
Elementary Student Survey
3. Student
a. Selects language [English or Spanish]
b. Enters assigned survey code:
Click down
For Spanish
4. When student finishes survey—in about
10 minutes—this final page should appear.
Program Staff & Teacher Surveys
Reports on Students in Afterschool Program
Program Staff and Classroom teachers are assigned
individual access codes
Use both individual access code and the ID code for each
student for whom they complete a report
Each student report takes 3-5 minutes to complete.
Student reports may be completed in multiple sessions.
For example: A staff completing reports for 20 students can do four
reports a day (a 20-minute session) on five different days.
Expands accountability tools beyond standardized test scores and
other academic measures
Provides a battery of valid, reliable and easy to administer youth
outcome measures
•
Currently include: Skill Development, Positive Behavior Change & Program Experience
•
Process of testing measures of STEM (Science, Technology, Engineering and Math)
learning in afterschool
•
Aim is to make available a menu of measures that match a range of student outcome
goals for diverse out-of-school time programs
Value in tracking individual student performance over time with
matched pre-post surveys—providing more powerful & interpretable
data
•
Matched = Respondent data is matched for both pre and post survey administrations (i.e.
same students, staff and/or teachers in fall and spring)
Information can be useful to to guide program improvement based on
student reports of program experience AND on student performance