IMPLEMENTING EVIDENCE-BASED PRACTICES

Prepared by the Justice Research and Statistics Association

THE IMPORTANCE OF EVIDENCE-BASED PROGRAMS

• • • The evidence-based movement is one of the most significant developments to occur in criminal and juvenile justice in recent years.

Evidence-based programs (EBPs) are important because they can be expected to produce results and a sound return on taxpayer investment.

Funding sources are increasingly promoting the use of EBPs.

Prepared by the Justice Research and Statistics Association 2

IMPLEMENTATION MATTERS

• Identifying and adopting an EBP is not enough to achieve successful outcomes.

• An evidence-based program still has to be delivered with fidelity/integrity in diverse and complex real world settings.

Prepared by the Justice Research and Statistics Association 3

HOW ARE EBPS IDENTIFIED?

• EBPs are deemed to be effective based on rigorous scientific research.

• While specific criteria may vary, all organizations assess the quality and consistency of evidence when determining program effectiveness.

Prepared by the Justice Research and Statistics Association 4

• • •

THE IMPORTANCE OF THE PROGRAM MODEL

Conclusions about a program’s effectiveness are derived from evaluations of a specific set of program components and activities, typically referred to as the program model.

An EBP should be implemented in a manner that is consistent with its program model.

Implementing an EBP in a way that deviates from the program model changes the program and invalidates expectations regarding the program’s effectiveness. Prepared by the Justice Research and Statistics Association 5

IMPLEMENTATION FIDELITY

• • • Implementation fidelity refers to the degree to which a program’s real-world implementation matches the program model.

Research has shown that programs implemented with a high degree of fidelity are more likely to be successful.

Deviating from the program model not only can degrade program effectiveness, but also can result in an otherwise effective program doing more harm than good.

Prepared by the Justice Research and Statistics Association 6

ORDINARY CIRCUMSTANCE CAN BE AN OBSTACLE TO IMPLEMENTATION

• • • What appears to be simple and straightforward in the implementation process often turns out to be more complex than anticipated.

We often underestimate the number of steps involved, the number of separate decisions that have to be made or the number of participants whose preferences have to be taken into account.

Because of the complexity involved, implementation, even under the best circumstances, is exceedingly difficult.

Prepared by the Justice Research and Statistics Association 7

HIGH-QUALITY IMPLEMENTATION IS DIFFICULT TO ACHIEVE

• • A 2011 Westat study found that only about 3.5% of all school-based programs to prevent youth substance abuse and school crime are research-based and well-implemented.

A 2013 EPISCenter study found that nearly half of the Blueprint program implementations in PA involved adaptation, and the majority (53%) of these adaptations were predicted to negatively impact program effectiveness.

Prepared by the Justice Research and Statistics Association 8

IMPLEMENTATION SCIENCE

National Implementation Research Network (NIRN)

Research-based insights and tools that can be used to support high-quality implementation in complex real-world settings Prepared by the Justice Research and Statistics Association 9

IMPLEMENTATION SCIENCE: LESSONS LEARNED

• • • Implementation is a process, not an event.

Implementation occurs in stages: – – – – Exploration: acquisition of information Installation: preparation, behind-the-scenes tasks Initial implementation: small scale, trial change in practice Full implementation: the innovation becomes accepted practice; full staffing and client loads Entire process takes 2-4 years.

Prepared by the Justice Research and Statistics Association 10

TRADITIONAL STRATEGIES TO PROMOTE EBPS ARE INSUFFICIENT TO ACHIEVE SOUND IMPLEMENTATION

• Diffusion and dissemination of information • Passing laws or using other mandates • Providing funding, or using funding incentives • Training • Organizational development Prepared by the Justice Research and Statistics Association 11

FACTORS THAT AFFECT IMPLEMENTATION FIDELITY

• Complexity of the program • Specificity of information in the program model • Facilitation strategies available to the implementing organization • Responsiveness of clients and staff Prepared by the Justice Research and Statistics Association 12

CORE PROGRAM COMPONENTS

• Core components are the parts of a program that are vital for program success and should not be eliminated or altered prior to or during implementation • When core components are known, organizations should ensure they are implemented with the highest degree of fidelity possible.

Prepared by the Justice Research and Statistics Association 13

ADAPTATION

• • The deliberate or accidental modification of a program – – – Deletions, additions or enhancements of program components Changes in the manner or intensity of administration of program components Cultural and other modifications required by local circumstances Excessive program adaptation is a common problem.

Prepared by the Justice Research and Statistics Association 14

ADAPTATION

• Finding the right balance between fidelity and adaptation is difficult.

• An evidence-based program should be implemented with the highest degree of fidelity possible.

• Adaptation is likely to be advantageous only when it is highly strategic, pursued with caution, and monitored to prevent potentially harmful effects.

Prepared by the Justice Research and Statistics Association 15

SYSTEMS TRUMP PROGRAMS

• Systems are not very accommodating to innovation and change.

• Systems will exert pressures to alter a new initiative so it fits into existing structures with the least amount of change. Prepared by the Justice Research and Statistics Association 16

IMPLEMENTATION DEPENDS ON:

• Core components of implementation that facilitate implementation fidelity • Based on the commonalities found among successfully implemented programs • Integrated and compensatory in nature Prepared by the Justice Research and Statistics Association 17

IMPLEMENTATION DRIVERS

• • • • • Staff selection Pre-service and in-service training On-the-job coaching Staff and program evaluation to identify problems and guide mid-course corrections Facilitative administrative support to address technical and adaptive problems Prepared by the Justice Research and Statistics Association 18

OVERCOMING RESISTANCE

• • •

Implementation teams

– – Must have the knowledge, skill, freedom, and authority to act within the larger organization or a collaboration of agencies Actively support and facilitate implementation on a daily basis

Communities of practice

– Mechanisms for practitioners to interact and exchange craft knowledge

Practice to policy feedback loops

Prepared by the Justice Research and Statistics Association 19

ASSESSING IMPLEMENTATION FIDELITY

• • Program structure – the framework for service delivery – – – – The presence of key program elements Participation of requisite partner organizations Resources allocated to the program Staffing levels and characteristics Program processes the way services are delivered – – – Adherence to program protocols or procedures Frequency and intensity of services provided The quality with which program staff delivers services Prepared by the Justice Research and Statistics Association 20

ASSESSING IMPLEMENTATION FIDELITY

• Context - prerequisites that must be in place for the program to operate – Staffing qualifications and numbers – – Practitioner-consumer ratio (i.e., caseload size) Completion of requisite training • Compliance with the intervention components prescribed in the program model • Competence or level of skill shown by the program staff Prepared by the Justice Research and Statistics Association 21

ASSESSING IMPLEMENTATION FIDELITY

• • • • Adherence to the program model - is the program being delivered as it was designed? Dosage or Exposure - do program participants receive the amount of the intervention prescribed in the program model? Quality of program delivery - do staff members deliver the program with skill?

Participant responsiveness - are program participants engaged by program activities?

Prepared by the Justice Research and Statistics Association 22