Transcript Using RE-AIM as a tool for Program Evaluation

Using RE-AIM as a tool for
Program Evaluation
From Research to Practice
What is RE-AIM
• RE-AIM is an acronym that consists of five
elements, or dimensions, that relate health
behavior interventions:
–
–
–
–
Reach the target population
Efficacy or effectiveness
Adoption by target settings or institutions
Implementation - consistency of delivery of
intervention
– Maintenance of intervention effects in individuals
and populations over time
• Commonly used for evaluating efficacy trials
(Phase III Research)
Defining Elements
Reach — The absolute number, proportion,
and representativeness of individuals who
participate in a given program.
Representativeness refers to whether
participants have characteristics that reflect
the target population's characteristics. For
example, if your intent is to increase physical
activity in sedentary people between the
ages of 35 and 70, you wouldn't test your
program on triathletes.
Defining Elements
Efficacy/Effectiveness — The impact of
an intervention on important outcomes.
This includes potential negative effects,
quality of life, and costs.
Defining Elements
Adoption — The absolute number,
proportion, and representativeness of
settings and staff who are willing to
offer a program.
Defining Elements
Implementation — At the setting level,
implementation refers to how closely
staff members follow the program that
the developers provide. This includes
consistency of delivery as intended and
the time and cost of the program.
Defining Elements
Maintenance — The extent to which a program
or policy becomes part of the routine
organizational practices and policies. Within
the RE-AIM framework, maintenance also
applies at the individual level.
At the individual level, maintenance refers to
the long-term effects of a program on
outcomes after 6 or more months after the
most recent intervention contact.
How do elements relate to
planning?
• As you design, plan, or evaluate a health
behavior intervention, there are questions
that you should ask yourself.
– Reach: HOW DO I REACH THE TARGETED
POPULATION FOR INTERVENTION?
– Efficacy or effectiveness: HOW DO I KNOW
THAT MY INTERVENTION IS EFFECTIVE?
– Adoption HOW DO I DEVELOP
ORGANIZATIONAL SUPPORT TO
DELIVERY THE INTERVENTION?
How do elements relate to
planning?
• As you design, plan, or evaluate a health
behavior intervention, there are questions
that you should ask yourself.
– Implementation: HOW DO I ENSURE THE
INTERVENTION IS DELIVERED PROPRLY?
– Maintenance: HOW DO I INCOPROATE
THE INTERVENTION SO IT IS DELIVERED
OVER THE LONG TERM?
How is RE-AIM different from
other evaluation approaches?
RE-AIM draws upon previous work in several areas
including:
Diffusion of innovations,
multi-level models, and
Precede-Proceed.
The primary ways that it is different is that it (a) is intended
specifically to facilitate translation of research to
practice, (b) it places equal emphasis on internal and
external validity issues and emphasizes
representativeness, and (c) it provides specific and
standard ways of measuring key factors involved in
evaluating potential for public health impact and
widespread application.
Is RE-AIM used to design programs,
or just to evaluate them?
• It is both. Although used more
commonly at present to report results
or compare interventions, it is also
intended as a planning tool.
An Example
• Physical Activity Promotion in Primary
Care: Bridging the Gap Between
Research and Practice
– Eakin, Brown, Marshall et al. (2004).
An Example of Application
Translating and Disseminating Evidencebased Falls Prevention Programs into
Community
Li et al. (2008). American Journal of Public Health
Primary Aim:
(a) To translate an evidence-based Tai Chi exercise fall
intervention into a community-based program for
implementation with older adults; and
(b) Using the RE-AIM framework (Glasgow et al., 1999), to
pilot implement the program with a primary focus on
reach, uptake (adoption), and implementation.
Secondary Aim:
• To evaluate the effectiveness of the program
with respect to improvements in physical
performance measures germane to falling.
Tertiary Aim:
• To evaluate program maintenance with
respect to the extent to which older adults
would continue to practice Tai Chi beyond the
period of the initial evaluation.
Translation
Translate an evidence-based Tai Chi exercise fall intervention
into a community-based program for implementation with
older adults
•
•
•
•
•
Identify training objectives and elements
Identify end users and dissemination partners
Develop a dissemination package
Expert evaluation
Pilot testing
Program Evaluation
Reach: A total of individuals/providers responded to the
program promotion
Effectiveness: defined as change in physical performance
outcome measures taken at baseline and again at 12
weeks termination.
Adoption (or uptake): defined as the proportion of local
community (senior activity) providers that agreed to
participate and implement the program.
Implementation: defined as the extent to which
providers' implemented a Tai Chi class to participating
older adults and the ability to conduct the various
elements of program protocols, including the use of
implementation plan, a 2-times weekly program
schedule, distribution of program supplements (i.e.,
videotape and a guidebook), a class attendance rate of
75% or better over the 12-week class period.
Maintenance: defined at both the service provider
level and participant level. At the provider’s level, it
was defined as providers’ willingness to consider the
program to be part of routine organizational provisions
(assuming adequate financial resources). At the
participant level, it was defined as the extent to which
improvements in participants’ physical performance
were sustained, and their continued practice of Tai Chi,
8-weeks after completion of the class.
Dissemination Outcome
REACH: 87% (by study criteria);
45% by client attendance
ADOPTION: Six senior activity centers
from five communities: 100% adoption
IMPLEMENTATION: 75% completed;
>85% class attendance; average 32 min.
of home practice
Dissemination Outcome
Effectiveness: Improved physical
performance and quality-of life measures
(a)Functional Reach; (b) Up and Go;
(c) Chair Stands, (d) 50-foot speed walk,
and SF-12.
Maintenances: Five centers continued;
87% participants continued
Impact
Adopted by the State of Oregon: being
implemented in four counties; three more
starting this year
Conclusion-Tai Chi
The evidence-based Tai Chi
program is practical to
disseminate and can be
effectively implemented
and maintained in
community settings