Transcript The Causal Effects of a School-Wide Social

School Selection and Randomization for a
School RCT of a Universal Social-Emotional
Learning and Literacy Intervention
Joshua L. Brown
Fordham University
Conference on National and International Perspectives on
Place-Based Randomized Trials in Education
Institute of Human Development and Social Change
New York University
October 3, 2008
NYC Study of Social and
Literacy Development
Principal Investigators
Joshua L. Brown
Fordham University
Stephanie M. Jones
Harvard University
J. Lawrence Aber
New York University
Acknowledgements
Research Team:
Genevieve Okada, Site Coordinator
Suzanne Elgendy, Vanessa Lyles, Emily Pressler, RAs
Wendy Hoglund, Postdoctoral Fellow
Maria LaRusso, Postdoctoral Fellow
Juliette Berg, Catalina Torrente, GRAs
Program Partners:
Tom Roderick
Audrey Major
Morningside Center for the Teaching of Social Responsibility
Funders:
Institute for Education Sciences, DOE
National Center for Injury Prevention and Control, CDC
William T. Grant Foundation
National Institute of Mental Health
Outline
1.
2.
3.
4.
5.
6.
7.
Background/Context of Study
Program and Study Design
Pre-Randomization Activities
Matching and Randomization
Implications of Approach
Current Preliminary Findings (Y1, Y1-2)
Conclusions and Future Challenges
Background (1)



Co-occurrence of social-emotional and
behavioral problems with low academic
achievement.
Theoretical and initial empirical links between
self-regulation and reading/math.
Emphases on standardized testing and
instructional improvement have crowded out
attention to social-emotional-character
development (among other things).
Background (2)



Early efforts at whole school strategies to
prevent behavior problems, violence, and
substance use plagued by intervention design
and implementation fidelity problems.
Early research on whole school strategies
plagued by low power, and inappropriate
statistical analyses.
Need to rigorously test promising but
unproven approaches to SEL/SACD.
Background (3)



Birth of “Social and Character Development”
Research Network.
7 different interventions in 7 different sites.
7 Local Evaluations and 1 National Evaluation
(Mathematica Policy Research).
Reading, Writing, Respect and Resolution (4Rs)
Program and Study Design
The 4Rs Program


Universal, school-based intervention in literacy development,
conflict resolution, and intergroup understanding.
3 Primary components:

7-unit literacy-based curriculum in conflict resolution and socialemotional learning.

Each unit organized around grade-appropriate book, includes 2
literacy activities, and 3-5 SEL lessons (21-35 total lessons).



Training and ongoing coaching of teachers in the delivery of the
4Rs curriculum.




Total possible activities per unit = 5-7
Total possible activities per year = 35-49
25 hours introductory training
Ongoing classroom coaching, minimum 12 contacts
Learning kit
Family Connections

1 parent-child “homework” per unit
Heuristic Model: 4Rs Child-Level Study
Teacher
Development
Social-Emotional
Skills &
Behaviors
4Rs
Experimental (classroom
and parent) vs.
Control
Extended
Opportunities &
Supports
Literacy Skills &
Academic
Achievement
Heuristic Model: 4Rs Setting-Level Study
School Culture and Climate
Teacher Affective
& Pedagogical
Processes &
Practices
4Rs:
Instruction,
Teacher Training
& Coaching
TeacherChild
Relationships
Child Behavioral
Dispositions &
Normative Beliefs
The Classroom System:
Culture and Climate
Classroom
Emotional,
Instruct. &
Org. Climate
Child
Developmental
Outcomes: SEL &
Academic Achievement
Overall Study Design





3-year, 6 wave longitudinal experimental design
18 NYC elementary schools matched and
randomly assigned to 4Rs and control group (9
assigned to each group)
Intervention is implemented school-wide, grades
K-6 for 3 years
All 3rd grade children in each school followed over
three years through 5th grade
Schools represent demographic character of NYC
public elementary schools

Racially/ethnically diverse; School lunch receipt ~70%;
Mobility/Stability = ~18%/60%; Suspensions = 23%
Pre-Randomization Activities
Identifying Candidate Schools
Planning Year (January-March, 2004)
 History of practitioner’s work in NYC led to letters of
support from Regional Superintendents
 Facilitated direct contact with Local Instructional
Superintendents who recommended schools based on:


no prior history of implementing 4Rs
willingness to implement program (and research) activities:




all Teachers participate in Intro. Training, teach curriculum ~1
lesson/week, and receive ongoing coaching from 4Rs Staff Dev’s
Principals attend 2-3 workshops/year and appoint “4Rs Liaison”
School administration and teachers cooperate with data collection
Resulted in LIS recommendations of 41 schools
Assessing Candidate Schools
Planning Year (March-June, 2004)
 Goal: assess/recruit “viable” schools for program
implementation in context of research study, (i.e.,
capacity for sustained, high-quality implementation, but
room to improve; willing to be randomly assigned)
 Process: Meetings and “walk-throughs” of all 41
schools:



Individual meeting with Principals to present program model
and overall research design
Visits to classrooms
“Organizational Readiness” assessment completed by
practitioners (co-developed with research team)
Organizational Readiness

Principal Leadership




Teachers & School Leadership Team




Organizational skills
Enthusiasm for/compatibility with 4Rs
Rapport with Students & Staff
Relationship with Principal
Enthusiasm for/quality of questioning about 4Rs
Stress, morale & attitudes toward children
School Environment



Tone of adult-child interaction
Engagement and behavior of students
Physical environment (e.g., use of bulletin boards, etc.)
Recruiting/Selecting Schools

17 schools eliminated from initial pool:


Grade structure other than K-5 (e.g., no 5th grade, K-3)
Lack of Principal and/or teacher interest in program




E.g., can’t support school-wide implementation requirement and/or
balance competing academic demands
Highly chaotic environments (e.g., adult-adult/adult-child
yelling, extensive behavior problems, frequent crises)
Unwilling to risk random-assignment to Control condition
(one preferred to purchase program)
24 schools held staff vote, signed letters of agreement
for random assignment
Pairwise Matching of Schools


Given potential for “bad draw” with small number of
schools, IES grantees agreed to pairwise match schools to
ensure balance on key variables and increase precision
24 schools pairwise matched and rank-ordered based on
“distance” of each school from every other eligible school
across 20 key school characteristics  12 pairs
School Characteristics Include:









Size (total N)
Race/ethnic and gender composition
School lunch receipt
Attendance (Students and Teachers)
Reading achievement (% of students at or above proficiency on ELA test)
Within year student mobility/two-year stability
Teacher full licensure and years of experience
Expenditures
Organizational Readiness (Overall)
Matching  Random Assignment



Funding for 18-schools, kept 9 best matching pairs, but
maintained 3 back-up pairs during lead-up to program
implementation/data collection
Random numbers generator used to assign 1 school in
each pair to intervention and 1 to control conditions
Post-random assignment, 2 schools and their respective
matches were dropped and 2 back-ups engaged



Principal of Tx school had been previously trained in RCCP
LIS ultimately did not condone RCT design for her schools
Note, pairwise matching can protect the experimental
design (from selection bias) in case of schools dropping
out after start of study (King et al., 2007)
Implications of Matching for Analyses


Should “blocks” (e.g., matched pairs) be regarded as fixed or
random effects?
Current debate in field, depends on:


Number of units per block (when only 2, need RE model)
Treatment effect heterogeneity (i.e, across matched pairs)


Interest in generalizability




If large, RE model allows heterogeneity of Tx effect to contribute to
standard errors and tests for the average effect of Tx.
In FE model, the blocks constitute the population or universe of
generalization
In RE model the blocks are seen as representing a larger universe of
possible blocks (or settings) in which Tx might be implemented.
See Schochet, 2004; Raudenbush, 2004; and Bloom, 2005;
all SACD internal network communications
We estimate blocks as random effects at the school-level –
most conservative
Implications of Approach





Evidence of effective matching process -- no Tx/Control
differences in 20 initial matching variables, or baseline
constructs assessed via child, teacher, and parent-reports
Focus on initial identification and subsequent selection of
“viable” schools limits generalizability to district- or
citywide population
However, features of school organizational capacity and
support have been clearly linked to schools’ ability for
quality program implementation (Payne, Gottfredson &
Gottfredson, 2006)
A fully representative sample of NYC elementary schools
would yield many schools with weak engagement and early
withdrawal from study
Although generalizable only to “willing” implementers, we
see preliminary evidence of Tx impacts
Year 1 Child Impacts
(Jones et al, under review)

Main effects of Tx on 2 of 9 child outcomes (2L HLM)


Controlling for baseline levels, children in the Tx group had
lower mean levels of Hostile Attribution Bias and Depression
than those in the control group at the end of Y1
Tx by baseline covariate interactions for 5 of 9 outcomes


E.g., Tx by baseline Behavioral Risk (“elevated” on teacherreport aggression and/or conduct problems at baseline)
Children with the highest level of baseline behavioral risk
show the greatest positive difference in Aggressive Fantasies,
Teacher-Report of Academic Skills, Reading Scale Score, and
Attendance between the intervention and control groups
Year 1 Classroom-Level Impacts

(in Effect Sizes)
Classrooms in the Tx group had higher mean Overall Classroom
Quality scores, accounted for by higher mean Emotional Support
and Instructional Support scores, than the control group
1
0.8
*
*
0.6
*
n.s.
0.4
0.2
0
Overall
Classroom
Quality
Emotional
Instructional
Organizational
Preliminary Year 1-2 Program Impacts
Child-Level
Results: Child-Level
TX Main Effects

Local: significant impacts for 2 of 6 constructs



Child Self-Report Hostile Attributional Biases
Child Self-Report Depression
Multisite: significant impacts for 3 constructs



Teacher-Report of Aggression
Teacher-Report Social Competence
Teacher-Report ADHD Symptoms
Y1-Y2 TX Main Effects: Summary
Intercept
Unstand. Est. (SE)
Slope
Unstand. Est. (SE)
Hostile Attributional
Bias
-.034 (.029)
-.043 (.023) t
Depression
.013 (.034)
-.065 (.023) *
TR: Social
Competence (Emotion
Regulation; Prosocial
Behavior)
-.009 (.100)
.137 (.077) *
TR: ADHD
(Hyperactivity;
Inattention)
.030 (.086)
-.084 (.053) t
TR: Child Aggression
.027 (.031)
-.047 (.017) *
TX on HAB Slope
0.54
Control
Hostile Attributional Bias (0-1)
Treatment
0.49
Control
0.44
Treatment
0.39
0.33
0
0.75
1.50
Time
2.25
3.00
Conclusions and Future Challenges

Trade-offs between generalizability and design feasibility
in school selection process for school RCTs.

We opted for selectivity based on:

LIS perspective of school need/capacity

Pre-randomization assessment by program practitioners of
school organizational readiness, used to identify final sample

Need rigorous and field efficient assessment tools that tap
multiple dimensions of school organizational capacity

How might RCT design innovations enable the inclusion
of disorganized and at-risk schools most in need of
intervention?