Transcript Document

Out of School Time in America
International Conference
Network on Extracurricular and Out-of-School Time
Educational Research
Denise Huang
CRESST/UCLA
November 23rd to 25th, 2010
Giessen, Germany
The Evolutions of Afterschool Programming for
At-Risk Youths
2 / 27
•
Safe Haven
•
Build Resiliency
•
Opportunities for Enrichment
•
Closing the Gap
•
Academic Achievement
THEORETICAL FRAMEWORK
3 / 27
A New Day For Learning
“A comprehensive , seamless approach to learning
that values the distinct experiences that families,
schools, after-school programs, and communities
provide for children.”
A Report from the Time, Learning, and
Afterschool Task Force
2007
4 / 27
The Ecological Model
Community/Neighborhood
School
Afterschool Program
Family
Child
Afterschool programs are social organizations embedded
with cultural values coming from families, school, and
neighborhood climate and norms.
5 / 27
Provisional Features of Positive
Developmental Settings (Larson et. al)








6 / 27
Physical and Psychological Safety
Clear and consistent structure
Supportive relationships
Opportunities to belong
Positive social norms
Support for efficacy and mattering
Opportunities for skill building
Integration of family, school, and community efforts
Core Positive Youth Development Constructs
Context
Person
Developmental
Success
Work
Place
Peers
Family
Congregation
Reduction
in HighRisk
Behaviors
Community
School
Programs
View of
the Child
Developmental
Strengths
Promotion of
Health Wellbeing
Thriving
Neighborhood
Search Institute Insights & Evidence, November 2006
7 / 27
Leveraging Social Capitals in
Afterschool Programs
Internal
Networks
Problem-solving
& Agility
Relationships
Social
Capital
Intellectual
Capital
Staff & Student
Relationships
Student
Engagement
8 / 27
Innovation and
Staff Development
Organizational
Capital
Competence
Attitudes
External
Networks
Trust
Expectations &
Social Norms
Theoretical Logic Model for the
Afterschool Partnership Study
Program Support
Management
Staff /Resources
Periodical Evaluation
for Continuous
Improvement
Content
After
School
Program
Quality
Set Goals
Content
Aligned
to Standards
Align
activities
to goals
Motivation
Engagement
Linkage to
School Day
Structure
9 / 27
Content
Practice
Quality
Opportunities
to Practice
(Attendance)
Research
Based
Practice &
Strategies
Process
Periodic
Assessment
of
Student
Performance
The CDE Indicator Model
Student
Engagement
Evaluation
System
Satisfaction
Monitoring
Goals
Management
Program
Orientation
CASHEE
Academic
External
Connections
Program
Climate
School
Attendance
Program
Environment
Relationships
Safety
Resources
Instructional
Features
Alignment
10 / 27
STAR
Staff
Efficacy
Positive
Youth Development
Setting Features
Expectation
Aspirations
Professional
Development
EVALUATION
11 / 27
The Need of Evidences to Support Expansions
 Do afterschool programs contribute to positive
academic development?
 Do afterschool programs contribute to positive
youth development?
 What aspects of the program functioning
contribute to these positive outcomes?
12 / 27
Prevalence of Different Evaluation
Research
Process Evaluations:
Formative studies
•
Evaluations that assesses the conduct of the program
during the initial design and testing stages with the intent
to improve the program
Program monitoring
•
Systematic examination of program coverage and delivery(target population, fidelity, efficiency)
•
Identifying successful implementation strategies for
program diffusion
13 / 27
Outcome Evaluation
Summative Evaluation
•
Summative evaluation provides information on the product's efficacy (
it's ability to do what it was designed to do)
•
By looking at the intervention group, the evaluator can examine the
learning materials and learning process together with the outcomes-hence the name Summative Evaluation.
Impact Evaluation
•
Impact evaluation involves constructing a counterfactual
•
Random selection and isolation from interventions are seldom
practicable and sometimes ethically difficult to defend.
•
Quasi-experimental method is often used.
14 / 27
Indicators for Program Effectiveness

Student attendance (in regular school and afterschool programs)

Performance measures (achievement tests, homework completion,
classroom grades, language re-designation, school retention, future
aspirations, etc.)

Non-cognitive measures (safety, attitudes towards school, relationships
with adults, social competence, conflict resolution skills, self-esteem,
and self-efficacy etc.)

Parent involvement

Professional development

Long-term effects (drop out rate, life satisfaction, etc)
15 / 27
Challenges in After School Studies

Diversity of program characteristics

Self-selectiveness
-Consent forms
 Comparison groups
-Ethical issues
 Transience
 Availability of archived data sources
 Meaningful outcome measures
 Importance of dosage
16 / 27
COGNITION & ACHIEVEMENT
17 / 27
Common Outcome Measures for Cognition
and Achievement
18 / 27

Academic achievement scores

Attitudes towards school/learning

Development of study skills (time management,
organization, memory, etc. )

Development of academic enabler (self-efficacy,
motivation, effort, etc.)

School dropout

Future aspirations
High Scope Afterschool Quality and Day
School Outcomes 2007
Safe
Environment
Positive
Interaction
Student
Engagement
Autonomy &
Opportunities to
reflect and practice
19 / 27
***
Reading
Achievement
Program
Attendance
School
Suspensions
Relationship between Program
Attendance and Youth Outcomes
Estimated Math Achievement Growth From
Baseline
0.100
0.080
0.060
0.040
LA's BEST
Attendance
0.020
0.000
Over 100 days
-0.020
51 to 100 days
-0.040
21 to 50 days
-0.060
0 to 20 days
-0.080
-0.100
2002-03
2003-04
2004-05
Time
20 / 27
2005-06
ASES Program Attendance
After School Program Attendance
Elementary Attendance (N=281,054)
Middle School Attendance (N=146,292)
30%
Percent of Students
25%
20%
15%
10%
5%
Days Attended
21 / 27
190-200
180-189
170-179
160-169
150-159
140-149
130-139
120-129
110-119
100-109
90-99
80-89
70-79
60-69
50-59
40-49
30-39
20-29
10-19
1-9
0%
22 / 27
Days Attended
190-200
180-189
170-179
160-169
150-159
140-149
130-139
120-129
110-119
100-109
90-99
80-89
70-79
60-69
50-59
40-49
30-39
20-29
10-19
1-9
Percent of Students
ASSETs Program Attendance
After School Program Attendance
High School Attendance (N=73,902)
70%
60%
50%
40%
30%
20%
10%
0%
Improved School Attendance and
Engagement in Learning
Elementary school students attending LA’s BEST afterschool program improved
their regular school day attendance and reported higher aspirations regarding
finishing school and going to college. Additionally, LA’s BEST participants are
20 percent less likely to drop out of school compared to matched
nonparticipants. (UCLA National Center for Research on Evaluation, Standards
and Student Testing, June 2000, December 2005 and September 2007)
A New Hampshire statewide study of students participating in academically
focused afterschool programs, including those funded by the federal 21st
Century Community Learning Centers Program (21st CCLC), found that more
than half of regular attendees improved both behaviorally and academically.
(RMC Research, 2005)
Ninety-two percent of high exposure of Citizen Schools participants were
promoted on time to the tenth grade compared to 81 percent of matched
nonparticipants. This is critical, since earning promotion to tenth grade on time
is a key predictor of high school graduation (i.e. preventing drop out). (Policy
Studies Associates, Inc., December 2006)
23 / 27
Improved School Attendance and
Engagement in Learning-High School
Ninth grade students who formerly participated in The After-School
Corporation (TASC) in middle schools had higher daily school
attendance and credit accumulation than matched nonparticipants.
(Policy Studies Associates, Inc., October 2007)
Participants in the Breakthrough Collaborative program enroll in
college-preparatory mathematics courses at double the national
average, and are accepted to college-preparatory high schools by
more than 80 percent. (Breakthrough Collaborative, 2006)
High school students participating in Chicago's After School Matters
program—which offers paid internships in the arts, technology,
sports, and communications to teenagers in some of the city's most
underserved schools—have higher class attendance, lower course
failures and higher graduation rates than similar students who do not
participate in the program. (University of Chicago, Chapin Hall Center
for Children, 2007)
24 / 27
Improved Test Scores and Grades
Annual performance report data from 21st CCLC grantees across the country demonstrate that students
attending 21st CCLC programs improve their reading (43%) and math grades (42%). (Learning Point
Associates, November 2007)
The Promising Afterschool Programs Study, a study of about 3,000 low-income, ethnically-diverse
elementary and middle school students, found that those who regularly attended high-quality programs
over two years demonstrated gains of up to 20 percentiles and 12 percentiles in standardized math test
scores respectively, compared to their peers who were routinely unsupervised during the afterschool
hours. (Policy Studies Associates, Inc., 2007)
Participants in North Carolina’s Young Scholars Program with at least 280 hours in the program
averaged double-digit increases annually for proficiency in both math and reading. Promotion rates rose
by 38 percent. Furthermore, the number of Young Scholars receiving A’s and B’s increased an average
of 38 percent, while the number receiving F’s decreased an average of 50 percent. (Z Smith Reynolds
Foundation, 2006)
Active participants in programs offered by The After-School Corporation (TASC) were more likely to take
and pass the Regents Math Sequential 1 exam by ninth grade than were nonparticipants. Thirty-two
percent of active ninth grade participants took and passed the exam, compared to one percent of ninth
grade nonparticipants. Fifty-two percent of active participants took and passed the Math Sequential 2
and 3 exams, compared to 15 percent of nonparticipants in the same grades. (Policy Studies Associates,
Inc., 2004)
Participants of St. Paul Minnesota’s 21st CCLC Pathways to Progress program received better grades in
English and math than nonparticipants. (University of Minnesota, Center for Applied Research and
Educational Improvement, March 2004)
25 / 27
Keeping Kids Safe and On Track for Success
A meta-analysis of 73 afterschool evaluations concluded that quality programs were consistently
successful in producing multiple benefits for youth including improvements in children's personal,
social and academic skills, as well as their self-esteem. (University of Illinois at Chicago,
Collaborative for Academic, Social, and Emotional Learning, 2007)
Children attending LA’s BEST Afterschool program are 30 percent less likely to participate in
criminal activities than their peers who do not attend the program. Researchers estimate that every
dollar invested in the LA’s BEST program saves the city $2.50 in crime-related costs. (UCLA
National Center for Research on Evaluation, Standards and Student Testing, September 2007)
Youth attending 23-40 or more days of Maryland’s After School Opportunity Fund Program
showed a more positive gain on commitment to education and academic performance, and a
reduction in delinquency and contact with the police. (University of Maryland, Department of
Criminology and Criminal Justice, June 2004)
Teens who do nothree times more likely to skip classes , use marijuana or other drugs, drink
alcohol, smoke cigarettes and engage in sexual activity.t participate in afterschool programs are
nearly (YMCA of the USA, March 2001)
26 / 27
Helping Families
Parents who are concerned about their children’s after-school care miss an
average of eight days of work per year. Decreased worker productivity related to
parental concerns about after-school care costs businesses up to $300 billion
per year. (Brandeis University, Community, Families and Work Program, 2004
and Catalyst & Brandeis University, December 2006)
In an evaluation of LA’s BEST, three quarters of the parents surveyed indicated
that since enrolling their children in the program, they worried significantly less
about their children’s safety and had more energy in the evening. (UCLA Center
for the Study of Evaluation, June 2000 and December 2005)
Parents in the TASC study said that the program helped them balance work and
family life: 94 percent said the program was convenient, 60 percent said they
missed less work than before because of the program, 59 percent said it
supported them in keeping their job, and 54 percent said it allowed them to work
more hours. (Policy Studies Associates, Inc., 2004)
After School Alliance: http://www.afterschoolalliance.org/after_out.cfm
27 / 27
Parents’ Perspectives
28 / 27
Parents’ Satisfaction
10
9
Mean Rating
8
7
6
5
4
3
2
1
The kinds of activites
offered
29 / 27
The overall performance of
afterschool staff
What my child learns in the
afterschool program
The materials and resources
the program provides for
parents
Obstacles for Parent Involvement



ASES
60% visited the
program
One in three
attended any event
One in five
volunteered or given
feedback
Conflicts
with job
30 / 27
Care for
other
children



ASSETs
One in five visited
the program
One in nine
attended any event
One in twenty
volunteered or given
feedback
Language
barrier
Lack of
transportation
PROGRAM QUALITY
31 / 27
Afterschool Accomplishments

Created a knowledge field that is in general in
consensus of program characteristics for high
quality indicators

Created a database now possible for investigating
longer term effects

Established a “nitch” and demonstrated the
significance/importance of ASP in educational
research
32 / 27
Common Practices of High Performing
Afterschool Programs
 Offering a broad array of enrichment activities
 Provide a wide range of
experiences that promote skill-
building and mastery
 Intentional relationship-building
 Employ strong managers, differentiated Staffing, and
 Partner Organizations Provided Support to Project Leaders
and Participants
Tasc, 2007
33 / 27
Successful Program Features
Best practices evidenced in the literature on
out-of-school time suggest that several
critical components such as:
• goal-oriented programs
• program structure
• and program process
These components contribute to the effectiveness
and success of programs.
Indicators for High Quality Programs
•
Program Structure
Clearly defined goals
Set up program structures to meet these goals
Set up program mission and vision to motivate staff
•
Process
Strong leadership
High quality staff
Clear communications and support
Positive relationships
•
Content
Research-based curriculum and strategies
Build in assessment and continuous improvement loop
35 / 27
Theory of Change
Making Programs Accountable and Making Sense of
Program Accountability
 Outcomes-based
 Causal model
 Articulate underlying assumptions
36 / 27
Evidence-based Teaching Approach-CAESL
 Sequenced - a sequenced set of activities to achieve skill
objectives
 Active - the use of active forms of learning
 Focused – program component to be focused on specific
skills
 Explicit – the targeting of specific skills
37 / 27
Use Internal Evaluation for Program
Improvement
 monitoring student progress
 measuring program growth
 researching program needs
 defining areas for professional
development
 gauging program impact
38 / 27
Model of Data-based Decision Making &
Continuous Improvement Process
Data Based
Decision-making
Process
Analysis of
Results
Assessment of
Student
Outcomes
Continuous
Adjustment
39 / 27
Plan of
Action
Specification
of monitoring
Pyramid of Program Quality – High Scope
Plan
Engaged
Learning
Make Choices
Reflect
Lead and Mentor
Opportunity for
Interaction
Supportive
Environment
Safety
40 / 27
Be in small Groups
Partner with adults
Experience belonging
Encouragement
Skill building
Active engagement
Psychological safety
Emergency procedures
Healthy food and drinks
Reframing conflict
Session flow
Welcoming atmosphere
Physically safe environment
Program space and equipment