Implementing Evidence-Based Programs in Systems of Care

Transcript Implementing Evidence-Based Programs in Systems of Care

OSEP 2006

Implementing Evidence-Based Education

Dean L. Fixsen, Ph.D.

National Implementation Research Network Louis de la Parte Florida Mental Health Institute

Education in the USA

In 1983, A Nation at Risk declared that American schools faced a “rising tide of mediocrity,” and that America was in danger of falling behind its international competitors because of the poor performance of its students.

National Commission on Excellence in Education (1983) More than 20 years later, the National Assessment of Educational Progress showed the achievement of U.S. students is virtually identical to what it was in the early 1980s.

Grigg, Daane, Jin, & Campbell (2003)

The Problem

Practices seem to be "influenced by fads and fashions that are adopted overenthusiastically, implemented inadequately, then discarded prematurely in favor of the latest trend.“ Walshe & Rundall (2001) reporting on health systems

The Problem

Human services are characterized by highly variable, often ineffective, and sometimes harmful services to consumers Institute of Medicine, 2001;2006; U.S. Department of Health and Human Services, 1999; 2001; New Freedom Commission, 2003 Reporting on health, mental health, substance abuse, emergency services

A Sobering Observation

"All organizations are designed, intentionally or unwittingly, to achieve precisely the results they get." R. Spencer Darling

Evidence-Based Programs

Innovations with strong evidence of effectiveness that affect core educational components:

•

instruction

•

curriculum

•

assessment

•

school organization Google: “EBE” 38,900 hits (June 2006)

Evidence-Based Programs

Improve student academic achievement significantly (e.g. Success for All: Slavin & Madden, 2002; Direct Instruction: Becker & Engelmann, 1995) Reduce referrals to special education by over 50% (e.g. Project ACHIEVE: Knoff, 2005) Reduce office discipline referrals by over 50% (e.g. School-Wide Positive Behavior Support: Sugai & Horner, 2002; Success for All: Slavin & Madden, 2002) Reduce high school drop out rates by 50% (e.g. STEP program: Felner et al., 2001) Reduce the incidence of childhood depression by over 50% (e.g. Kam, Greenberg & Kusche 2004)

Evidence-Based Programs

Considerable research suggests minority students and disadvantaged students are particularly sensitive to the quality of instruction they receive. EBE can significantly improve outcomes for these students.

(e.g. Slavin & Madden, 2002)

Evidence-Based Programs

Rhetoric: No Child Left Behind and Comprehensive School Reform (Obey-Porter) legislation exhort educators to use “scientifically proven” methods Reality: Most federal funding is for “innovations” and “reforms” that are home grown or variations of traditional curricula; very few “innovations” are based on experimental research of any sort (Slavin, 2005)

What is missing?

RESEARCH

EDUCATION

The Problem

Science to Education Gap What is known is not what is adopted to help children learn Implementation Gap What is adopted is not used with fidelity and good effect What is implemented disappears with time and turnover

Follow Through Programs

Figure 1: This figure shows the average effects of nine Follow Through models on measures of basic skills (word knowledge, spelling, language, and

math computation), cognitive-conceptual skills (reading comprehension, math concepts, and math problem solving) and self-concept. This figure is

adapted from Engelmann, S. and Carnine, D. (1982), Theory of Instruction: Principles and applications. New York: Irvington Press.

Implementation

Establish new ways of functioning in education Beyond the rhetoric, how can education systems be changed?

Implement Innovations

The act of accomplishing some aim or executing some order To put into practical effect; carry out Pursue to a conclusion – Dictionary.com

Implementation of innovations with benefits to students.

Teaching-Family Model (1967 - )

Teaching –Family Replications

The First 794 Attempted Replications (1967 – 1982) Fixsen, Blase, Timbers, & Wolf (2001)

300 1974 1976 1978 1980 1982 250 200 150 100 50 0 Cumulative Homes Cumulative Couples 16

The NIRN

Craft knowledge EBP purveyors (program developers) EBP implementation site managers Implementation researchers Survey of EBP program developers Scientific information CRUSK, Follow Through, Lodges, et al.

Program development and replication data Synthesis of the implementation evaluation and research literature

Implement Innovations

Effective education practices

+

Effective implementation practices

= Good outcomes for students

Implement Innovations

Effective IMPLEMENTATION Effective NOT Effective Benefits to Students NOT Effective

Implement Innovations

An evidence-based program is one thing Implementation of an evidence based program is a very different thing

Implementation What has been tried?

Implementation

Excellent experimental evidence for what does not work Dissemination of information by itself does not lead to successful implementation (research literature, mailings, promulgation of practice guidelines

)

Ev Rogers’ summaries of diffusion research are well diffused but an adoption decision is not implementation

Implementation

Excellent experimental evidence for what does not work Training alone, no matter how well done, does not lead to successful implementation With a variety of designs and measures, information dissemination and training by themselves repeatedly have been shown to be ineffective in education, human services, health, business, and manufacturing

Implementation

Excellent evidence for what does not work Implementation by edict does not work Implementation by “following the money ” does not work Implementation without changing supporting roles and functions does not work Paul Nutt (2002). Why Decisions Fail

Implementation Success What works?

Stages of Implementation

Implementation is not an event A mission-oriented process involving multiple decisions, actions, and corrections

Wexelblatt’s Scheduling Algorithm

CHEAP FAST GOOD When implementing an innovation, you may pick any two.

Stages of Implementation

Implementation occurs in stages: Exploration Installation Initial Implementation Full Implementation Innovation Sustainability 2 – 4 Years

Implementation Framework

School, District, State Structures/Culture Teachers, Staff, Schools Purveyor Evidence-Based Programs Fidelity & Outcome Measures

Fixsen, Naoom, Blase, Friedman, & Wallace, 2005

Evidence-based programs

The usability of a program has little to do with the quality or weight of the evidence regarding that program Evidence about intervention effectiveness for specific populations helps us choose what to implement Evidence about the effectiveness of the intervention does not help implement the program or practice successfully

Evidence-based programs

Key aspects of core components Clearly described (what/how) Practical measure of fidelity Fully operationalized (do/say) Field tested (iterative revision) Contextualized (school/systems fit) Good outcomes (worth replicating)

Teachers, Staff, Schools

Teachers impact students

It is the job of superintendents, principals, and funders to align policies and structures to facilitate effective teacher practices There is no such thing as an “administrative decision” – they are all education decisions

Active Purveyor Role

Successful implementation on a useful scale requires a purveyor A group of individuals thoroughly familiar with a program who actively work to implement that program with fidelity and good effect Purveyors accumulate data & experiential knowledge



more effective and efficient over time (information economics, K. Arrow)

Active Purveyor Role

Change the behavior of adult education professionals Change organizational structures, cultures, and climates Change the thinking of system directors and policy makers Successful and sustainable implementation of evidence-based practices and programs always requires organizational change .

Active Purveyor Role

Simultaneous, Multi-Level Interventions Teacher School Management (leadership, policy) Administration (HR, structure) Supervision (nature, content) District State and Community Context

Real World Implementation

Ready Fire Aim Service systems are moving targets.

A process, not an event.

Implementation Drivers

STAFF PERFORMANCE EVALUATION CONSULTATION & COACHING DECISION SUPPORT DATA SYSTEMS INTEGRATED & COMPENSATORY PRESERVICE TRAINING FACILITATIVE ADMINISTRATIVE SUPPORTS RECRUITMENT AND SELECTION SYSTEMS INTERVENTIONS

TRAINING COMPONENTS Theory and Discussion ..+Demonstration in Training …+ Practice & Feedback in Training …+ Coaching in Classroom OUTCOMES (% of Participants who Demonstrate Knowledge, Demonstrate new Skills in a Training Setting, and Use new Skills in the Classroom) Knowledge Skill Demonstration Use in the Classroom 10% 30% 60% 95% 5% 20% 60% 95% 0% 0% 5% 95% Joyce and Showers, 2002

Systems Transformation “Systems trump programs.”

Patrick McCarthy, Annie E. Casey Foundation

Systems Transformation

Evidence-based programs are the leading edge of Education System Change Restructure instructional, administrative, and policy environments to enable education systems that are: More dynamic and adaptable More accountable More effective

Systems Transformation

Initiate & Manage Change Implement Innovations Sustainable Infrastructure

Policy - Practice - Feedback

Policy (Plan) Policy Structure Procedure Practice (Do) Practice Form follows Function

Implementation Essentials

Help faculty/staff acquire the knowledge, skills, and abilities to effectively provide innovations to students Help schools/districts change to more effectively support the work of the faculty/staff Transform education systems to effectively facilitate and sustain the use of innovations statewide

Education Implementation

What Works Clearinghouse http://www.whatworks.ed.gov/ National Dissemination Center for Children with Disabilities http://www.nichcy.org/ Council for Excellence in Government http://www.excelgov.org

Center for Implementing Technology in Education http://www.citeducation.org/ National Association of State Directors of Special Education, Inc. http://nasdse.org/

Thank You

We thank the following for their support Annie E. Casey Foundation (EBPs and cultural competence) William T. Grant Foundation (implementation literature review) Substance Abuse and Mental Health Services Administration (SOC analyses of implementation; implementation strategies grants; NREPP reviews; implementation awards) Centers for Disease Control (implementation research contract) National Institute of Mental Health (research and training grants) Juvenile Justice and Delinquency Prevention (program development and evaluation grants)

For More Information

Karen A. Blase

813-974-4463 [email protected]

Dean L. Fixsen

813-974-4446 [email protected]

National Implementation Research Network At the Louis de la Parte Florida Mental Health Institute University of South Florida

http://nirn.fmhi.usf.edu

For More Information

Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M. & Wallace, F. (2005).

Implementation Research: A Synthesis of the Literature.

Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231).

E-mail us for a free copy or download all or part of the monograph at:

http://nirn.fmhi.usf.edu/resources/publications/Mono graph/index.cfm