Transcript Coping with Violence: School-Based Research with Urban

Adaptation and Fidelity
In the Implementation of
Evidence-Based Programs
Emily Ozer, Ph.D.
University of California, Berkeley
School of Public Health
[email protected]
For Today
• Give overview of research program
and motivation for it
• Describe recent study relevant to
violence prevention
• Answer any questions re career
development, methods, etc
• PLEASE “Interrupt”
Roots of Research Interests
• Human rights and community-based work
– India - Community-based health promotion, research
– Central America - Human rights, impact of violence
– California – Democratic business with Latina
immigrant women and families
• Informs research on:
– Complex interventions in schools and communities -linking process, context, and outcomes
– Individual and collective “resilience” to range of
stressors – violence, extreme poverty
Research on Factors Promoting
Mental and Physical Health
• Protective factors for psychological problems and
aggression in adolescents who have experienced
community violence (e.g. Ozer & Weinstein, 2004, Journal of Clinical Child
and Adol Psych; Ozer, 2005 J. of Adolescent Research)
• Perpetration of violence in dating and peer contexts
among Mexican and European American adolescents (Ozer,
et al., 2005, J of Adolescent Health)
• Risk and protective factors for PTSD and other stressrelated problems across the lifespan
– Meta-analysis of risk factors for PTSD (Ozer et al. Psych Bull 2004)
– Child mental health in context of domestic violence (Lieberman, VanHorn, & Ozer,
Development and Psychopathology 2006)
– Oportunidades: Impact on child behavior problems (Ozer et al. Pediatrics, 2009)
• Resilience-oriented; factors amenable to intervention*
Population Health:
School-Based Interventions
• Peer-led HIV prevention: How peer educator-student
relationships and classroom context predict program
outcomes (Ozer, Maslach, Weinstein, et al. 1996, American Journal of Community
Psych)
• Contextual factors that influence effectiveness of schoolbased violence prevention (e.g. Ozer, 2006, J of Primary Prevention)
• School gardens as a model for strengthening school
setting and improving child/adolescent health (Ozer, 2006,
Health Educ and Behavior)
• Sources of school connection – qual follow up to exposure
to violence study (Ozer, 2008, Journal of Adolescent Research).
Community-Based Interventions:
Focus on Urban Public Schools
• Excellent way to reach diverse youth and
families from under-served communities
• Critical setting for youth development
– Strengthen resources in setting through
research
– Ultimate goal of school-level change
Community-Based Interventions:
Opening up the “Black Box”
• Beyond overall effects, fine-grained
intervention research:
– For whom and under what conditions is the
intervention more effective?
• Specify theory of change
• Operationalize intervention processes and
contexts expected to promote change
• Link intervention processes and outcomes
– Implications for theory and policy
Overview of Current Research
• Two related studies that develop and
test models with adolescents as:
– “Agents” and resources in secondary
school settings
– Collaborators in efforts to improve
adolescent health and wellbeing
• Bridging participatory research and traditional
intervention research – longstanding interest
Overview of WT Grant Scholars’
Award Research
• Developing and testing models of teens as
collaborators in urban public schools
– Youth-led participatory action research
– Adaption/fidelity of evidence-based programs
• What happens when diffused? Are teacher and
student adaptations consistent with program theory?
• Ozer et al. in press Prev. Science
• “Stretch” Areas
– Mixed methods, qual, quant
Adaptation/Fidelity Study Rationale
• Competing pressures in diffusion of
evidence-based prevention programs
• High-fidelity implementation of programs
• Involvement of local stakeholders, community
boards
• Little study of integration of local expertise and
evidence-based programs
Central Questions and Goals
Initial Phase of Research
• How programs “fit” local settings when
diffused?
• Eliciting views of local stakeholders,
especially youth, on fit with norms, practices
in schools, communities
• Characterize if suggestions support core
components or not
• Longer term goal: Test added value of
collaborative implementation
Specific Questions Addressed Today
• What changes do teachers make when
implementing evidence-based prevention
programs, and why?
• What kinds of adaptations do teachers vs.
youth suggest regarding evidence-based
prevention programs?
• Are these suggested changes consistent
with program theory?
Site and Teacher Selection
• Two urban school districts – Oakland, SF
• Diverse student populations (ethnicity,
immigration status, and socio-economic status)
• “Higher end” of implementation spectrum
• Roll-out of evidence-based programs in high
schools
• Understudied; few programs; likely more adaptation
• Collaborative process developmentally-appropriate
• District nominated health teachers likely to
do high-quality job – make curriculum focus
Prevention Programs
• Project Towards No Drug Abuse (TND)
– 12-session curriculum
– Theoretical model focusing on
• “Motivation-skills-decision-making”
• Building social and self-control skills
• Strengthening rational decision-making processes
• Too Good for Drugs and Violence (TGDV)
– 14-session curriculum
– Theory of change drawing from range of theories, e.g.
• Social norms
• Social cognitive theory
• Social development model
Multiple Sources of Data
•
Classroom Observations (163)
•
•
•
Majority of sessions observed in each class
Student Group Interviews (25 interviews)
Teacher Interviews (22 formal)
•
•
•
Report of any adaptations made to curriculum
Rationale for making any adaptations
Suggested adaptations to the curriculum
Data Sources:
Program Developer Consultations
• Asked program developers to characterize
each suggested change:
• How consistent with core program theory and
teaching approaches?
• Integrated these responses into qualitative
dataset of suggested adaptations
Qualitative Analysis
• Coded by at least two members of the research team
• Discrepancies resolved by discussion in weekly
meetings and revision of the codebook
• Codes entered in Nvivo qualitative software program
• Observed Adaptations: Field notes and teacher
interviews coded for observed or reported adaptations
• Suggested Adaptations: Interviews with teachers and
students coded for adaptation suggestions
Sensitizing Concepts Based on
Prior Conceptual Frameworks
• Frameworks for cultural tailoring (e.g.
Resnicow; Kreuter)
• Surface/peripheral
• Deep/socio-cultural
• Evidential
• Others emerged from collected data
• Content, real life, resources, teaching
approach
Coding of Feedback From
Program Developers
• “Stoplight” categories (CDC, ETR)
– Theoretical, logistical “stoplight” code for each
suggested adaptation
– Green, yellow, red
Overview of Results
• Analysis of changes made by teachers
• Analysis of teachers’ and students’
suggested adaptations to curriculum
• What types of changes suggested?
• Consistency between teachers and
students?
• Acceptability to program developers?
Results: Teachers’
Observed Adaptations
• What changes did teachers make?
– Changes in instructional format (45%)
– Integration of real-life experiences of the teacher or
others into the curriculum (21%)
– Supplementation with additional resources such those
used before (16%)
• And why?
– Primarily pedagogical reasons
• Experiential learning; addition of visual, kinesthetic materials
– Classroom dynamics and management
• E.g. “unsafe” to do stereotype activity in racially charged school
– Time and sequencing
• E.g. Already taught communication skills in health class
Results: Teachers’
Observed Adaptations
• Do teachers differ in adaptations in 1st vs. 2nd
time teaching the curriculum?
– Overall, same number of adaptations observed…
– But 2nd time more adaptations in teaching and
resources (χ2= 3.97, df = 1,89, p < .05; χ2= 7.25, df = 1,89, p <
.01)
• Parallels teachers’ comments about “following the
script” the 1st time, then “making it their own.”
• Tension between fidelity and effective teaching.
Results: Students’
Suggested Adaptations
• Content: More information (29%)
– e.g. “positive aspects of drug use”
• Surface: Wording, appearance (26%)
– e.g. “no one says ‘dweeb.’”
• Teaching approach: (20%)
– e.g. making the curriculum more interactive by
including more games or group activities
Results: Teachers’
Suggested Adaptations:
•
Instructional practices such as enacting
experiential activities and adding homework to
deepen students’ reflection on topics (34%)
•
More information, particularly about
different types of drugs and their effects (16%)
•
Surface aspects of the curriculum such as
language and including more visuals (13%)
Acceptability of Teachers’ and
Students’ Suggestions
• Theoretical consistency: Equally likely to
propose “green-light” changes to each
program.
• Logistical concerns: Suggestions equally
likely to be judged acceptable or problematic
by program developers.
Type of Adaptation
by Acceptability: TND
•
Some students’ suggestions “red lighted” due to
concerns about deviancy training or scare tactics
– Outside speakers talk about addiction
– More differentiated characters and trajectories
• Some teachers’ suggestions were “red lighted”
– Replaced empirically-tested activities with
those of questionable effectiveness
• e.g. Demo of drug effects
– Detracted from main goals of session
Type of Adaptation
by Acceptability: TGDV
• Small number of suggestions “red-lighted”
– Main concerns
• Covering content re intervening in fight
• Dropping of an activity
• Any discussion that might deviate from “zero
tolerance” of fighting and drug use
– Students say you have to fight back to save face
Discussion and Next Steps:
Actual Changes
• All teachers made changes
• Tended to make more interactive to address different
learning styles
• Contrast to prior observational study by Hansen
• Concerns about knowledge transmission
• State standards are not skills-based
• Next steps:
• Deeper qualitative inquiry into tensions across
practice communities, meaning of fidelity
• Engaging stakeholders in dialogue about meaning of
fidelity (“scalpel vs. chainsaw”)
Discussion and Next Steps:
Suggested Changes
• Some differences in type of suggestions made
by teachers versus students
• But youth equally likely to suggest “green light”
• Many surface suggestions could increase perceived
relevance without sacrificing fidelity to program core
• Next steps
• Comparative evaluation of regular versus adapted
• Program developers’ integration of changes into new
versions of curricula
Youth-Led Participatory Research
• Youth participatory/action research in urban
secondary schools
– What are core features and processes?
– Impact on youth participants?
– Impact on school settings?
• Extensive development
– Collaboration, capacity (Ozer et al. 2008, AJCP)
– Measures of key processes and outcomes (Ozer
et al. in press, AJCP, Ozer & Schotland, in press, HEB)
What is Youth-Led
Action Research?
• Iterative process of inquiry and action led
by youth and guided by adult allies:
– Identify problems and questions
– Conduct research (with training) to guide
recommendations and solutions
– Engage and advise decision-makers
• Recent topics: Diversity, drop-out, stress, school
lunch, teachers’ best practices for culturally
responsive teaching, sexual health, dating
relationships
YPAR Study: Within-School Multi-Method Design
4 diverse sites x 5 years: Randomized at
Classroom Level
“Peers” Youth-Led
Action Research
Elective Class
N= (135)
Latino (34%)
Asian American (37%)
African American (28%)
Pre/Post/FU Surveys,
Classroom Observations,
Interviews
“Peers” Mentoring
Direct Service
Elective Class
Other Electives
N= (171)
Latino (25%)
Asian Am (52%)
African American (6%)
Latino (48%)
Asian Am (42%)
African American (8%)
Pre/Post/FU Surveys,
Classroom Observations,
Interviews
N= (240)
Pre/Post/FU Surveys
(measure development)
Figure 3. Youth Research Intervention: Study Design and Sample
Intervention
PAR
Key Processes
Classroom
Youth-level
Outcomes
Teacher-student
power-sharing
Skills, efficacy in research,
communication, advocacy
Networking
opportunities
Positive ethnic identity,
Sense of purpose
Group work
Perceived school
connection
Positive class climate
(e.g. engagement,
student perspectives)
Opportunities for
skill development
(e.g. research, advocacy)
Psychological and political
empowerment
Expanded social networks
and support
Figure 1a. School-Based PAR: Youth-Level Effects Model
Settings Targeted
School-Level
Outcomes
Alliances between
students and adult staff
Classroom
Expansion of meaningful student
roles -school policies and practices
PAR
School
Student-adult
inquiry and learning
Figure 1b. School-Based PAR: School-Level Effects Model
Sample Thus Far
• Baseline: 24 experimental classes; 23
experimental control
• Challenges of small class sizes, attrition,
and semester vs. year long structure
– BL: experimental=135, exp control=171
– 3 mos: exp=130, exp control=154
• 22 quasi-experimental control classes
Individual-Level Analyses
• Cluster-randomized structure
• Challenges
– Likely different effects across sites
– Small samples
– Different expert advice, “cultures”
• Current analyses in STATA with robust SE
adjusted for clustering
Individual Preliminary Findings
• In the midst of analyses….
• Higher dosage associated with:
– Participatory behavior
– Motivation to influence
– Self-esteem
School Setting Analysis
• Developmental mismatch in public
secondary schools (e.g. Eccles, Midgley, et al.)
• Recent attention to characteristics of
settings that promote positive development
– Ways of intervening to improve settings
• YPAR as potential means to expand
meaningful participation, efficacy,
“mattering”
Design + Sample
• Does YPAR process expand opportunities
for influence, participation, “mattering?”
– Definitely within the class. On level of school?
• How experienced by participants?
– Organize around critical incidents/interactions?
• What conditions set the stage for new or
expanded interactions and roles?
• Which if any are sustained over time?
Mentors and Collaborators for
YPAR Research Project
Mentors: Tom Cook (Northwestern), Meredith
Minkler (UCB), Larry Green (UCSF)
School Collaborators: Elizabeth Hubbard, Gary
Cruz, Adee Horn, Morgan Wallace
Research team: Marieka Schotland, Laura
Douglas, Sami Newlan, Dana Wright, Maggie
Gaddis, Miranda Ritterman, Yolanda Anyon, Eddy
Jara, Kathryn Clark, Huong Dao, Eric Koo, Becky
Lee, Christopher Wu
Funding: WT Grant Foundation, CDC