Scaling Up Evidence-based Practices Dean Fixsen, Karen Blase, Rob Horner, and George Sugai Current Challenges Effective practices exist for literacy, social behavior and safety Traditional TA approaches (dissemination.
Download ReportTranscript Scaling Up Evidence-based Practices Dean Fixsen, Karen Blase, Rob Horner, and George Sugai Current Challenges Effective practices exist for literacy, social behavior and safety Traditional TA approaches (dissemination.
Scaling Up Evidence-based Practices Dean Fixsen, Karen Blase, Rob Horner, and George Sugai Current Challenges Effective practices exist for literacy, social behavior and safety Traditional TA approaches (dissemination and compliance) are insufficient Methods are available to reliably produce benefits to students statewide © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 Currently EBPs now are boutique operations ◦ Now have convincing demonstrations that EBPs can work in the real world ◦ This is encouraging, but EBPs are not used on a sustainable scale sufficient to solve social problems © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 Future Start with the end in mind What will it take to: ◦ make statewide use of education innovations ◦ that produce increasingly effective outcomes ◦ for the next 50 years? © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 Problem Most States already have attempted to implement a variety of these science-based practices Some attempts achieve their purposes, but others fall short © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 Key Issues Four big issues have emerged: ◦ Select what to scale up ◦ Determine how to implement new education practices so they actually produce the intended benefits to students (effective) ◦ Define how to scale up effective practices so they are available to all students who could benefit from them (effective, accessible) ◦ Define how to align system structures and functions to fully support scale up efforts as part of “education as usual” (effective, accessible, sustainable) © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 When is a “practice” ready for TA Investment Defining a “practice” ◦ A “practice” is a procedure, or set of procedures, designed for use in a specific context, by individuals with certain skills/features, to produce specific outcomes for specific individuals. Operationally defined procedures Target population/ Context Implementer Characteristics Defined outcomes The scope/size of a practice matters Practices can be small, medium or large in scope ◦ Time delay ◦ Functional behavioral assessment ◦ First Step to Success Scale up practices only when they are bundled to a level where they have impact on a core social outcome. © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 When is a “practice” ready for TA Investment Wisdom from: Karin Guldbrandsson H. Gotham Dean Fixsen and Karen Blase George Sugai Bruce Joyce and Beverly Showers Eight Features of a “practice” that make it “ready for implementation” investment When is a “practice” ready for TA Investment Substantive impact on highly valued outcome. Practice is comprehensive enough that it addresses a highly valued outcome (not just a piece) There is a perceived need for the outcome Effect is dramatic (both in relative and absolute terms) The practice is defined with precision The practice can be described with operational precision and substance. What is the “it” Compatibility with existing practices When is a “practice” ready for TA Investment Evidence-based Experimental documentation of relationship between practiced and impact published in peerreviewed journal. (Effect Size of at least .50) Emerging Promising Evidence-based Efficiency Minimal additional resources to perform new practice Minimal additional resources to transition to new practice When is a “practice” ready for TA Investment Social Validity The people required to behave differently are comfortable with the new behavior(s) they are being asked to perform Documentation of effectiveness trials Adaptability The practice can be adapted to “fit” different contexts The challenge of being precise and manualized versus being flexible and adaptive (be clear about core). Generalizability The practice is effective across the array of contexts normally encountered. Priority Effectiveness Valued Outcomes DataBased Prob. Solving Continuous Regeneration Identifying & Modifying Practices Practice Implementation Efficiency What “packaging features” are necessary for successful adoption? Materials for exploration and awareness What is the need, what is the practice, what is the documented link between the practice and valued outcome? Materials for Installation and Initial Implementation Materials for teaching people to master the new practice Protocol for implementation What “package features” are necessary for successful adoption? Evaluation tools Is practice being implemented with fidelity Is practice producing valued outcome Are training segments producing implementation of core elements of “practice” Research Measures Universal Tier of -School-wide Evaluation Tool SWPBS (SET) Secondary and Tertiary Tiers of SWPBS Overall Summary of Implementation -Individual Student Schoolwide Evaluation Tool (I-SSET) Annual SelfAssessment Measures -Self-Assessment Survey (SAS) -Benchmarks of Quality (BoQ) Progress Monitoring Measures -Team Implementation Checklist (TIC) -Benchmarks of Advanced Tiers (BAT) (To be developed) -Implementation Phases Inventory (IPI) -Phases of Implementation (POI) © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 Evaluation Guided Action Planning Activity 1. Establish Commitment Administrator Top 3 goal 80% of faculty Three year timeline 2. Establish Team Representative Administrator Effective team operating procedures Activity Task Analysis a. b. c. d. e. Who When a. b. c. d. e. Audit of teams/initiatives © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 Key Issues Interventions that are and remain effective through several generations of teachers, principals, superintendents, and State and national leaders Implementation supports that are and remain effective through several generations of trainers, coaches, evaluators, administrators, and State leaders © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 Teaching–Family Replications Fixsen, Blase, Timbers, & Wolf (2001) 900 300 800 700 250 600 200 500 150 400 300 100 200 Cumulative Homes Cumulative Couples 50 100 0 0 ≤1972 1974 1976 1978 1980 1982 © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 Follow Through Programs Figure 1: This figure shows the average effects of nine Follow Through models on measures of basic skills (word knowledge, spelling, language, and math computation), cognitive-conceptual skills (reading comprehension, math concepts, and math problem solving) and self-concept. This figure is adapted from Engelmann, S. and Carnine, D. (1982), Theory of Instruction: Principles and applications. New York: Irvington Press. © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 School Adopting School-wide Positive Behavior Support 10000 9000 8000 7000 6000 5000 4000 3000 2000 1000 0 03-04' 04-05' 05-06' 06-07' 07-08' 08-09' Insufficient Methods Excellent experimental evidence for what does not work Diffusion/dissemination of information by itself does not lead to successful implementation (research literature, mailings, promulgation of practice guidelines) Training alone, no matter how well done, does not lead to successful implementation © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 Insufficient Methods Excellent experimental evidence for what does not work Implementation by laws/ compliance by itself does not work Implementation by “following the money” by itself does not work Implementation without changing supporting roles and functions does not work Paul Nutt (2002). Why Decisions Fail © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 Reduction in Incidence of Mental Retardation and Learning Disabilities The Oregon Department of Education has released graduation rates for all public high schools. Sobering Observation Nearly one-third of all high school students don't receive a diploma [and systems] after four years of study. are "All organizations designed, intentionally or unwittingly, by Betsy Hammond, Riseto in Incidence of Autism achieve precisely the results they The Oregonian Monday June 29, 2009, get." R. Spencer Darling Business Expert © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 Sobering Observations Innovative practices do not fare well in old organizational structures and systems Organizational and system changes are essential to successful implementation ◦ Expect it ◦ Plan for it © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 Benefits to focusing on Scaling Implementation Build on local strengths, context and accomplishments Implement 1-3 evidence-based practices at a sustainable scale of social importance. Document value of implementation to achieve valued student outcomes © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 Benefits to focusing on Scaling Implementation Develop capacity of your State to implement any evidence-based practice with fidelity and good outcomes De-silo implementation of evidencebased practices, innovations, legislative mandates in the State © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 Scale Up To scale up interventions we must first scale up implementation capacity Building implementation capacity is essential to maximizing the use of EBPs and other innovations Large scale, real time change © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 Multi-tier Model Academic Systems Intensive, Individual Interventions •Individual Students •Assessment-based •High Intensity •Of longer duration Targeted Group Interventions •Some students (at-risk) •High efficiency •Rapid response Universal Interventions •All students •Preventive, proactive 80-90% Behavioral Systems 1-5% 1-5% Attention, 5-10% Effort, 5-10% Precision 80-90% Intensive, Individual Interventions •Individual Students •Assessment-based •Intense, durable procedures Targeted Group Interventions •Some students (at-risk) •High efficiency •Rapid response Universal Interventions •All settings, all students •Preventive, proactive Capacity Development State Management Team State Transformation Team Regional Implementation Teams © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 SISEP Support & 2 FTE IMPLEMENTATION CAPACITY FOR SCALING UP EBPs State Management Group State Transformation Team Regional Implementation Team Regional Implementation Team N = 50 – 200 Schools N = 50 – 200 Schools Regional Regional Implementation Implementation Team Team N = 50 – 200 Schools N = 50 – 200 Schools First Regional Implementation Team N = 9 Staff Practice Informed Policy (PIP) System Change Policy Enabled Practice (PEP) SISEP Support & 2 FTE State Management Group Context Many initiatives to improve education Initiatives too often conflicting and/or competing Wraparound Math Using RTI to Align Systems Primary Prevention Early Intervention Multi-tiered Support Wraparound Early Math Intervention Family Support Behavior Support Student Outcomes © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 Response to ALIGNMENT Literacy Universal Screening Progress Monitoring Systems to support practices Implementation Team A group that knows the innovation very well (formal and practice knowledge) A group that knows how to implement that innovation with fidelity and good effect A group that accumulates data & experiential knowledge -- more effective and efficient over time (information economics, K. Arrow) © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 Implementation Team Prepare Communities Prepare Districts Prepare schools faculty, staff Implementation Team Work with Researchers © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 Assure Student Benefits Assure Implementation Capacity Development Start with too many overqualified people “Generation 1” RIT members become: ◦ Generation 2 expanded RIT members ◦ Generation 3 State Transformation Team leaders ◦ Generation 4 State and federal officials © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 IMPLEMENTATION CAPACITY FOR SCALING UP EBPs State Management Group [Phase 2] SISEP Support State Transformation Team 2 FTE & Others Regional Implementation Team Regional Implementation Team N = 50 – 200 Schools N = 50 – 200 Schools Regional Regional Implementation Implementation Team Team N = 50 – 200 Schools N = 50 – 200 Schools State Capacity Development Regional Impl. Teams 16 14 RITs 12 10 SaturationSaturation 44/51 8 Intensive Development 6 Intensive Development 4 STTs 2 0 1 2 3 4 5 6 7 Years 8 9 10 11 12 Coaching Defined Coaching is the active and iterative delivery of: ◦ (a) prompts that increase successful behavior, and ◦ (b) corrections that decrease unsuccessful behavior. ◦ Coaching is done by someone with credibility and experience with the target skill(s) ◦ Coaching is done on-site, in real time ◦ Coaching is done after initial training ◦ Coaching is done repeatedly (e.g. monthly) ◦ Coaching intensity is adjusted to need Outcomes of Coaching Fluency with trained skills Adaptation of trained concepts/skills to local contexts and challenges And new challenges that arise Rapid redirection from miss-applications Increased fidelity of overall implementation Improved sustainability Most often due to ability to increase coaching intensity at critical points in time. Training Outcomes Related to Training Components Training Outcomes Training Components Presentation/ Lecture Knowledge of Content Skill Implementation Classroom Application 10% 5% 0% Plus Demonstration 30% 20% 0% Plus Practice 60% 60% 5% Plus Coaching/ Admin Support Data Feedback 95% 95% 95% Joyce & Showers, 2002 Coaching within SWPBS Implementation Context: ◦ 9600 schools implementing SWPBS nationally Defining the Role Internal vs External Selecting Coaches Training and support for coaches Assessing Impact Rehearsal and Coaching After initial training, a majority of participants (211 of 213) demonstrated poor implementation. Decision-makers should pair training prior to implementation with on-going rehearsal and performance feedback (coaching) ◦ Test et al 2008 © Dean Fixsen, Karen Blase, Robert Horner, George Sugai, 2008 Avg. Referrals per Day Example of the Impact of Coaching on Student Outcomes: Average Major Discipline Referrals per Day per Month Coach returns from leave 7 6 5 4 3 2 1 0 Sep 05-06 Oct 06-07 Nov Dec Jan Feb Mar Apr May Visibility Funding Political Support Leadership Team Active Coordination Training Coaching Behavioral Expertise Local Demonstration Schools Evaluation “Even if you’re on the right track, you’ll get run over if you just sit there” - Will Rogers