Transcript Document

The California Statewide ASES Evaluation
CERA Conference
November 18th, 2010
San Diego, California
Purpose of Study
1. Prop 49 becomes effective in Sept 2006
2. Afterschool funding increases to 550 million annually
3. One of the stipulation is that the CDE shall contract for an
Independent Statewide evaluation on the effectiveness of
programs receiving funds.
4. CRESST will conduct 2 separate evaluations- ASES and ASSET
5. Evaluation will cover analyzing STAR and other CDE data
from 2006-2007 through 2009-2010
2 / 27
Deliverables
3 / 27
•
Annual reports for ASES & ASSETs including
achievement and behavioral outcomes
•
Annual reports on afterschool profiles and changes
•
In year 3 and 4 of the study, youth development
outcomes will also be included in the annual reports.
•
Final report to the Governor and the Legislature on
December 1st 2011.
Study Theory
Best practices evidenced in the literature on
out-of-school time suggest that several
critical components such as:
• goal-oriented programs
• program structure
• and program process
These components contribute to the effectiveness
and success of programs.
Theoretical model
Student
Engagement
Evaluation
system
Satisfaction
Monitoring
Goals
Management
Program
Orientation
CASHEE
Academic
External
Connections
STAR
School
Attendance
Program
Climate
Program
Environment
Relationships
Positive
Youth Development
Safety
Resources
Instructional
Features
Alignment
5 / 27
Staff Efficacy
Setting
Features
Expectation
Aspirations
Professional
Development
Data Sources
CDE will provide:
•
STAR
•
CAHSEE
•
CELDT
•
SSID maintenance system
•
Graduation rate for High School students
•
Annual accountability reports with student level data, including
attendance in school and afterschool
6 / 27
Data Collected by CRESST
•
Profiling survey distributed to all grantees
 program goals, structures, and process
•
In depth site visits
 program satisfaction, program process, community
partnerships, program impacts
7 / 27
Study Sample Definition
•
Study Sample I
All programs participating in the ASES and ASSET programs
•
Study Sample II
100 ASES and 30 ASSET districts
•
Study Sample III
All programs returning the profiling questions
•
Study sample IV
Small sample for in depth analyses
8 / 27
The 6 evaluation questions for Study Sample IV
1.
What is the impact of after school programs on the academic performance of
participating students? Does participation in after school programs appear
to contribute to improved academic achievement?
2.
Does participation in after school programs affect other behaviors such as:
school day attendance, homework completion, positive behavior, skill
development, and healthy youth development?
3.
Examine the similarities and differences in program structure and
implementation. Describe how and why implementation has varied across
programs and schools, and what impact these variations have had on
program participation, student achievement, and behavior change.
4.
What is the level of student, parent, staff, and administration satisfaction
concerning the implementation and impact of after school programs?
5.
What is the nature and impact of organizations involved in local
partnerships?
6.
What unintended consequences have resulted from the implementation of
the after school programs?
9 / 27
Preliminary Afterschool Attendance Findings
•
Elementary students tend to attend more regularly
•
About 50% attended over 100 days
•
Middle school students less regularly
•
26% less than 10 days, about 20% attended 100 or more
days
10 / 27
Preliminary School Attendance Outcomes
•
Statistically significant effect on School
attendance
•
Effect higher for frequent attendees
•
No difference on school suspension-Sample II
11 / 27
Program Characteristics
•
Most Programs are free for all
•
First come first serve
•
Majority of the programs stated that they have an academic
goal
•
Top three programming activities offered are:
homework assistance, academic enrichment and tutoring
•
Top three programming activities offered are:
physical fitness/sports, arts/music, and recreation
•
Top developmental activities offered are:
12 / 27
school safety, youth development
ASES
Example: Goal Setting & Progress
Set goal
93.6%
Met goal
15.4%
Progressed
towards
goal
49.1%
Improved day school
attendance (n = 275)
68.0%
27.8%
36.9%
34.8%
0.5%
Improved homework
completion (n = 279)
80.6%
29.3%
40.9%
29.3%
0.4%
Positive behavior
change (n = 270)
69.3%
19.8%
50.8%
28.3%
1.1%
Improved program
attendance (n = 274)
86.9%
43.7%
35.3%
19.3%
1.7%
Increased skill
development (n = 260)
62.3%
23.5%
45.1%
30.2%
1.2%
Goal
Academic
improvement (n = 298)
13 / 27
Set goal, but
not
evaluated
35.5%
Failed
to
progress
0.0%
Professional Development
•
Site coordinators are the most likely to receive
professional development
•
The three top format are: training/workshop,
program level meetings, and site level meetings
•
The top three topics are:
classroom management, behavioral management, and federal
mandated trainings
14 / 27
Program Orientation
•
Most programs use a collaborative management approach
•
Having prior work experienced is important
•
Most site claimed to have at least some credentialed staff on
site at any day—especially in the urban areas
•
Support of educational goals is the most likely recruitment
strategy for ASES programs
•
Staff recognition and is the most mentioned strategies for
retaining staff, followed by promotion opportunities
15 / 27
Community Partnership
•
Majority of the programs maintained community partnership
•
ASES programs partnered with parents, school and district staff and
high school students
•
The entity most school sites partnered with is the local LEAs of
school districts or colleges
•
LEA helps in providing operational assistance (data collection), site
management, and professional developments
•
Parents and other community members generally were involved in
setting and revising goals, providing goods and helping out with the
implementation and fundraising of the programs.
•
Other community members also played a role in fundraising for the
programs.
16 / 27
Additional Insights from
Preliminary Findings of Sample IV
•
Students perceived the program climate to be safe physically and
emotionally. They felt cared for and supported by the staff, while
the staff reported to encourage students’ efficacy and aspiration.
•
In general, staff reported to have sufficient writing materials, space
to work, and enough time to work with the students. Both site staff
and site coordinators were likely to report that they did not have
time to prepare for activities.
•
staff efficacy was enhanced by professional development. Both site
coordinators and site staff agreed that the professional
development offered was aligned with program goals and that the
training was provided annually.
•
Parent surveys reviewed a low degree of involvement, though they
also appeared to be satisfied with the way programs connected with
them.
17 / 27
Evaluation Systems
•
Most site coordinators reported that they tracked the
different stakeholders’ levels of satisfaction
•
They also responded that they had an internal method of
evaluating the program activities for program improvement
18 / 27
Perceived Outcomes
•
Overall, students, parents, and after school staff all have positive
perceptions on the effects of program participation. Both students
and their parents felt that attending a program led to students
feeling more academically efficacious. They also believed
participation increased the students’ cognitive competence, & socioemotional competence.
•
Students reported that they felt that they had the skills, knowledge,
and desire to become successful adults as a result of attending an
after school program.
•
High school students further reported that attending a program
helped them resist the lure of alcohol, drugs, and gangs.
19 / 27
Obstacles for Parent Involvement
ASES
Conflicts with
job
20 / 27
•
60% visited the program
•
One in three attended any event
•
One in five volunteered or given
feedback
Care for other
children
Language
barrier
Lack of
transportation
Obstacles to Student Recruitment and Retention
ASES-NONE
Staff
Language barrier
Lack of transportation
Student
Other activities
21 / 27