Transcript Prevention Programs that Work: Evidence

Implementing and Evaluating
Evidence-based Strategies
Where do We Stand?
Assessing
your
Community
Establishing
Goals &
Objectives;
Planning for
Evaluation
Finding
Evidence
Selecting Best
Fitting EBAs
Adapting
Implementing &
Evaluating
Session Objectives
• Describe successful program implementation
• Discuss the importance of community
engagement, partnerships, and collaborations to
implement programs
• Describe key tasks in planning, implementation,
evaluation and maintenance
• Develop an implementation and evaluation work
plan integrating RE-AIM measures that match
program objectives and activities
Intervention Planning Model
Community
Assessment
Share your
successes
***
Apply lessons
learned to
Maintain and
Sustain the
intervention
Develop
Goals &
Objectives
Plan the Intervention
Finding, Selecting and Adapting
an Evidence-based Approach
Assess Readiness
Evaluate Processes
Evaluate
Outcomes
Implement
Planning Steps
• Enlist community/stakeholder input
– Use existing committees
– Understand current context, policy or environment
– Best communication/outreach strategies
• Understand context, fit, and your readiness
– Incorporate previous needs assessment data
– Know your organizational culture related to health
• Draft work plan for program, policy, strategies
– Research any previous interventions
for useful content/language
– Receive feedback from expert panel
Planning your implementation
and evaluation: Review Materials
Policy Process Guides
http://www.prevent.org/data/files/initiatives/smokefreepolicies.pdf
Assess Readiness
• Do you have:
– Supportive leaders?
– Engaged partners?
– Resources (money, staff, supplies)?
– Staff with knowledge, skills, and enthusiasm?
– Accounting and administrative structures?
– System for monitoring progress/evaluation?
• Refer to implementation guides and protocols for
resources needed
Assess Organizational Readiness
Readiness Checklist for Body and Soul
What leads to implementation quality?
1. Community level engagement
(funding, policy, politics)
2. Provider/implementer characteristics
– Need for innovation
– Benefits of innovation
– Self-efficacy
– Skill proficiency for implementation
Durlak, J.A., & DuPre, E.P. (2008) Implementation matters: A review of research on the influence of
implementation of program outcomes and the factors affecting implementation. Am J Com Psych. 41:327-350.
What leads to implementation quality?
3. Implementers’ organizational capacity
– General organizational factors
(vision, norms, climate)
– Specific processes
(decision-making, partnerships, communication)
– Staff considerations
(leadership, champion)
4. Prevention support
– Training
– Technical assistance
Durlak, J.A., & DuPre, E.P. (2008) Implementation matters: A review of research on the influence of
implementation of program outcomes and the factors affecting implementation. Am J Com Psych. 41:327-350.
Create a Work Plan
Ideas from your
Organizational Readiness
Assessment
Your Work Plan for
Implementing and Evaluating
Body and Soul
Components of a Work Plan for
Implementation
•
•
•
•
•
•
SMART Objectives
List of tasks or activities for each Objective
Person responsible
Resources needed
Target completion dates
Indicator(s)/Measures of completion or progress
Example: Work Plan for Body and Soul
Example Work Plan
Example Work Plan
Definitions of Workplan Steps:
Policy & Environmental Change
Formulation
• Review evidence/data on the problem and potential solutions
• Gain stakeholder agreement on priority problems and
preferred approaches,
• Draft policies in the form of rules/laws
Enactment
• Identify decision makers who will sponsor or promote the policy
• Formally pass the policy in the form of a law, regulation, or budget
• Put environmental change in place (ex: inaugurate walking trail)
• Communicate the new policy/environmental change
• Monitor and document the use of the policy
Implementation
Maintenance
• Continue promotion, operationalization, and use of the policy
• Conduct activities that support compliance to the policy
• Establish a plan for enforcement
• Maintain policy implementation
• Ensure sustained effect through monitoring and enforcement
• Possibly make further policy modification
Longest Jr. BB. (2009) Health Policymaking in the United States, 5th Ed.
Activity: Create an
Implementation Work Plan
Activity: Create an
Implementation Work Plan
Activity: Create an
Implementation Work Plan
1. Review the handout on the Tobacco Use
Prevention Policy Strategy
2. Choose at least 1-2 of the objectives already
provided or write your own (for a worksite or faith
setting)
3. For the objective(s) you chose, add:
–
–
–
–
–
One or more activities
Who would be responsible for each activity?
What resources are needed?
What is the completion date or due date?
What are your measures/indicators of completion or
progress?
Implementation Work Plan
Discussion
4. Share different examples from the groups.
a) Specify if you chose a work site or faith setting.
b) Explain which objective you chose.
c) Discuss:
– activities,
– responsible persons,
– resources,
– dates, and
– measures of completion or progress
EVALUATION Work plans
have many of the same elements
• SMART Objectives
• Activities for each Objective
• Indicators/Measures (of completion or progress)
• Person responsible for evaluation activities
• Resources needed
Data sources
• Target completion dates
Actual completion dates
• Processes for monitoring, reporting, modifying
implementation and evaluation plans
If you are adopting an existing evidencebased program, policy or other strategy...
• Conduct formative evaluation on adapted materials
– Feedback from expert panel
– Focus groups/discussion with target populations
– Pilot testing
• If it was evaluated before, you can use prior process
and outcome indicators, methods, and tools
– What outcomes were measured?
– How rigorous was the original evaluation?
– Do you have resources to do a similar evaluation?
• Monitor fidelity of implementing core elements and key
process steps (process evaluation)
Evaluation – Using RE-AIM
•
•
•
•
REACH your intended target population
EFFECTIVENESS or efficacy
ADOPTION by target staff, settings, or institutions
IMPLEMENTATION consistency, costs and
adaptations made during delivery
• MAINTENANCE of intervention effects in
individuals and settings over time
http://www.re-aim.org/
Evaluation Phases
Planning
•Consult
with the program stakeholders or
resources
•Develop the questions
•Identify sources of data
•Make a timeline
Data
Collection
•Pilot
test
•How will the questions be asked?
•Who will ask them?
Data
Analysis
•Who
Reporting
•Who
will analyze the data and how?
will report and how?
•Who will receive the data and when?
•How will it affect the program?
Gather Credible Evidence
Evaluation
questions
Indicators
Methods
and
Measures
(not the other way around)
Gather Credible Evidence
• Link each indicator to a data source
• Look for existing data
– If there are no data available – collect your own
• Look for existing data collection tools
– If there are no existing tools – create your own
• Data collection plan
– Who will collect?
– How will it be collected?
– When and How often will it be collected?
Evaluation for Programs
• Track program promotion and/or recruitment
(# flyers, frequency of announcements)
• Document implementation of program core elements or
key process steps
(e.g., checklists, length & # of sessions, meetings)
• Collect process measures
(e.g., attendance, timeline/work plan updates, etc.)
• Monitor program activities and performance
(# of new screenings, timeliness of diagnosis)
Evaluation for Policies
• Document and publicize policy
• Track implementation of key process steps of the policy
–
–
–
–
Documentation of policy
Procedures for policy
Communication
Enforcement
• Collect process measures on benchmarks
–
–
–
–
Track participants in policy planning (partners)
Revisions of policies
Advocacy efforts (e.g., media outlets, events)
Adoption and enforcement
Example Evaluation Plan: Body and Soul
Example Evaluation Plan: Body and Soul
Discussion: Develop an Evaluation Plan
Discussion:
Develop an Evaluation Plan
1. Refer to the SMART objective you chose in the
work plan activity.
2. Discuss evaluation question(s) that measure
success of that objective.
3. What are potential data sources to find this
information?
4. What are some methods you can use to
measure these processes or outcomes?
(e.g. logs, surveys, etc.)
Intervention Planning Model
Community
Assessment
Share your
successes
***
Apply lessons
learned to
Maintain and
Sustain the
intervention
Develop
Goals &
Objectives
Plan the Intervention
Finding, Selecting and Adapting
an Evidence-based Approach
Assess Readiness
Evaluate Processes
Evaluate
Outcomes
Implement
Share Evaluation Results
• Use your evaluation findings
– to modify your program
– describe program benefits and challenges
• Reach all stakeholders inside and outside
your organization
– In language and formats that everyone
understands
– In time to make decisions
Maintenance…Plan for Sustainability
Sustainability is…
… the existence of structures and processes which
allow a program to leverage resources to most
effectively implement evidence-based policies and
activities.
Luke D, Calhoun A, Herbers S. (Feb 2012). Building & Managing Sustainability in Public Health
Programs. CDC Roundtable Presentation.
Maintenance Activities
Leverage resources to sustain interventions over time:
• Provide referrals for further service, when needed
• Seek additional funding
• Maintain or secure a program champion
• Make program a part of organizational services/policies
• Surveillance of the impact over time
(e.g., tracking of smoking rates, BMI of children, screening)
• Demonstrate return on investment/cost savings to justify
investment of other stakeholders
• Continue partnerships or build new ones to share costs
and implementation delivery
Sustainability Planning Resources
https://sustaintool.org/
Schell, S.F., Luke, D.A., Schooley, M.W., Elliott, M.B., Herbers, S.H., Mueller, N.B., & Bunger, A.C. (2013). Public health program
capacity for sustainability: A new framework. Implementation Science, 8(15).
Take-home Points: Importance of
Implementation and Evaluation
• Successful implementation and evaluation requires
detailed, advanced planning
• Involve partners from the beginning to implement with
high quality
• Assess organizational readiness and build capacity
• Use and adapt pre-existing guidance and tools
• RE-AIM can be helpful to measure success in
meeting objectives
• Use evaluation findings and lessons learned to plan
for long-term sustainability
Questions?