Transcript Prevention Programs that Work: Evidence

Planning for Evaluation
Where do We Stand?
Assessing
your
Community
Establishing
Goals &
Objectives;
Planning for
Evaluation
Finding
Evidence
Selecting Best
Fitting EBAs
Adapting
Implementing &
Evaluating
Session Objectives
• Define different types of evaluation
• Explain the difference between process
and outcome evaluation questions
• Identify measurable outcomes that are
linked to each program objective
• Use methods that match program
objectives and activities
Questions
• Why evaluate?
• Why NOT evaluate?
• Which do you begin first?
– Implementing your program?
– Implementing your evaluation?
Evaluation helps you to…
• Communicate findings to staff, partners and
funders
• Share with other practitioners what you did, how
you did, and what did/did not work for you
• When findings indicate that objectives were not
met, explore and address the reasons
• When findings indicate objectives were met,
celebrate!
Find balance between outcomes
and feasible methods
• What outcomes are most important?
• What evaluation methods are most
feasible?
Engaging Stakeholders
• Who are the stakeholders for your
programs?
• What questions and outcomes
matter most to them?
Evaluation Types
• Process evaluation
– Are all activities being implemented as planned?
– Are you reaching the intended audience?
• Short-term outcome evaluation
– Does behavior, knowledge or attitude change?
– Is a policy documented? Is it enforced?
• Long-term outcome evaluation
– Does health improve?
– Do morbidity and mortality decrease?
You can measure success if…
…there are clear, measurable indicators of
success (SMART objectives)
– Delivery, participation in, attitudes about, or
uptake of a program (Process)
– Health behavior change (Short-term Outcome)
– Policy, practice or environmental change
(Short-term Outcome)
– Health change (Long-term Outcome)
Process Evaluation
• Can find problems early and show if progress is on course
• Includes measurement of adoption, reach, participation,
exposure, implementation (fidelity), satisfaction, barriers
and facilitators, and maintenance
• Data collection methods may include:
–
–
–
–
–
–
activity tracking databases/work plans
implementation logs/fidelity checklists
registration sheets
observation
participant satisfaction surveys
key informant interviews to improve initiatives (debriefing)
Windsor, R.A., Clark, N.M., Boyd, N.R., & Goodman, R.M. (2003) Evaluation of health
promotion, health education, and disease prevention programs
Causes of Program Success or Failure
The design of the intervention
Sound
Faulty
Sound Program success Intervention problem
The way the
intervention is
implemented
Implementation
Faulty problem
Program failure
Using the RE-AIM Framework
The RE-AIM framework is designed to enhance the quality,
speed, and public health impact of efforts to translate
research into practice in five steps:
• REACH to your intended target population
• EFFECTIVENESS or efficacy
• ADOPTION by target staff, settings, or institutions
• IMPLEMENTATION consistency, costs and
adaptations made during delivery
• MAINTENANCE of intervention effects in
individuals and settings over time
http://cancercontrol.cancer.gov/is/reaim/index.html
http://www.center-trt.org/index.cfm?fa=webtraining.reaim
Short-term Outcome Evaluation
• Measures progress in meeting your objectives
• Assesses awareness, knowledge, attitudes, behavior,
skills, changes in a setting or environment
• Can assess enforcement of a policy (including in schools,
worksites, faith organizations, etc.)
• Can include environmental changes (e.g. increased
access to healthier foods)
• Data collection methods may include:
– pre/post surveys
– interviews, focus groups about beliefs and behavior change
– observation of events or environment/infrastructure
Long-term Outcome Evaluation
• Measures long-term effects such as changes in:
– Morbidity
– Mortality
– Quality of life
-
Cost effectiveness
Behavior change
Environment/system policy changes
• Can assess if conditions improved in the target population
and if improvements were equitably distributed
• Data collection methods may include:
– Registries
– Surveillance data collection
– Longitudinal data collection over extended time
– Surveys, interviews, focus groups, observation
Example: Body and Soul Program
Core Elements
Body & Soul is an evidence-based program for African
American churches to promote consumption of more fruits
and vegetables. There are 6 program core elements:
•
•
•
•
•
•
Project Committee
Kick-off event
At least 3 church-wide nutrition events
At least 1 additional event involving the pastor
At least 1 church food habit change (policy change)
2 motivational counseling calls by volunteers to each
participant
Evaluation Methods: Body & Soul Replication
Type of Evaluation
Evaluation Questions
Process Evaluation
Reach: How many churches and members
participated?
Implementation: Did you implement all six core
elements?
Implementation: What were reactions to the
program?
Short term
Outcome
Evaluation
What changes occurred?
Adoption:
Were church policies adopted for healthy
eating?
Long term
Outcome
Evaluation
Effectiveness:
-Did members eat more fruits and vegetables?
-What changes in health outcomes occurred?
(e.g. reduced rates of chronic disease,
improved quality of life, etc.)
Activity: Planning for Evaluation Developing Evaluation Questions
Planning for Evaluation Activity
Instructions
1. A worksite is planning a smoke-free policy for interior and
exterior spaces. The policy will not permit smoking of any
kind in the building, stairwells, entrance ways, green space
or parking areas.
2. In groups of 3-4, first discuss how you envision the smokefree policy playing out at a generic worksite. Then, write 3-4
evaluation questions to measure process, short-term, and
long-term outcomes.
3. Use RE-AIM to help you write questions that address
different aspects of policy intervention.
4. Work on this exercise for about 15 minutes.
Evaluation  Next Steps
Discussion:
• What kind of information do your current
stakeholders want to know?
• What types of evaluation could fulfill those
questions?
• How could evaluation planning help guide
what strategies you ultimately select and
implement?
Take Home Points: Importance
of Evaluation
• Evaluation planning begins during the planning
phase
• Engage stakeholders to help you determine what
outcomes are most important and what are the
most feasible ways to measure each outcome
• Create evaluation questions that have measurable
outcomes and that are linked to each objective
• Measure process outcomes, as well as short and
long-term outcomes
Questions?