Evaluation Overview-

Download Report

Transcript Evaluation Overview-

Evidence-Based Public Health: A
Course in Chronic Disease
Prevention
MODULE 8:
Developing an Action Plan and
Implementing Interventions
Darcy Scharff
March 2012
Learning Objectives
1. Identify key characteristics and principles in
successful action planning.
2. Understand the purpose and use of logic models.
3. Describe steps used in constructing logic models.
2
Learning Objectives
1. Identify key characteristics and principles in
successful action planning, including the role of
coalitions/ partnerships.
3
Action versus ongoing planning
 Action planning
for a defined program or policy with specific, time
dependent outcomes
 Ongoing or strategic planning
a regular function within an organization
4
Importance
Changeability
5
Key characteristics of effective
action planning
 Plans that clearly spell out and make linkages
across
goal
objectives
action strategies
 Roles and responsibilities of important
stakeholders clarified (timetables, training)
 Clear mechanisms for tracking progress
(evaluation)
 Action plans based on evidence and
assessment
6
In planning, remember
to link evidence with programs
Data
Interpretation
Data
Analysis
Data
Collection
Program
Evaluation
Information
Dissemination
Program
Implementation
Program
Planning
7
Key Principles of Planning
 Assessment should guide the development of
programs
 Community members should participate in the
process
 Program should be developed with key partners or
as part of coalitions (builds infrastructure for change)
 Participants should develop a comprehensive
intervention strategy- but may implement one step at
a time
8
Key Principles of Planning (cont)
 Evaluation should emphasize feedback and program
improvement
 The community capacity for health promoting changes
should be increased – skill transfer between
community & academic/practice partners (two way)
9
Aspects of Effective Coalitions
 Build a sense of community
 Enhance community member engagement –
mechanisms for true involvement
 Strong leadership – decision making, conflict
management
 Provide vehicle for community empowerment
 Diversity valued and celebrated
 Incubator for innovative solutions to large problems flexible and responsive
10
Aspects of Failed Coalitions
 Individuals/organizations have different expectations
of coalition
 No clear direction or goals
 Ineffective or unchanging leadership
 No involvement of those effected by the programs/
policies
 Focus on unrealistic, distant outcomes
11
Learning Objectives
1. Identify key characteristics of successful
action planning
2. Understand the purpose and use of logic
models
12
Logic model: Definition
Diagram depicting interrelationships between
 goal -longer term public health outcomes
 objectives -shorter term intervention impacts
 action strategies
Also called
 Analytic framework
 Causal frameworks
13
Purpose of logic model
 Aids in program planning
 Guides steps of data collection
 Defines evaluation process
Map to linkages on which to base conclusion about
intervention effectiveness
Linkages represent ‘causal pathways’
 Identifies primary and intermediate effects
14
Inputs
Activities
Outputs
Outcomes
Short-Term
Educational
classes
Financial
Resources
Human
Resources
Informational
Resources
Newsletters
Media campaign
Workshops
Seminars
Coalition
building,
advocacy, etc.
Completed
Activities
Examples:
• # of people
participating in
each event
•# of newsletters
distributed
•Amount of air
time or number
of media events
• # of
workshops/semin
ars conducted
• # letters
• # meetings
Increased
knowledge, skills
and abilities
Intermediate
Increase in physical
activity and healthy
eating (behavior)
Long-Term
Decrease in
Obesity
Goal
Reduce type-2
diabetes
Logic Model (Analytic Framework) Worksheet: Evidence-Based Public Health
Program Title:__________________________________________
Goal:
Long-Term Objective:
What are the evidencebased determinants?
Intermediate Objective
(Environmental Level):
Activities:
Activities:
Activities:
Activities:
Costs:
Costs:
Costs:
Costs:
Evaluation
Intermediate
Objective (Govt./Org.
Level):
Process
How much will it cost?
What other resources
are needed
d?
Intermediate
Objective (Social
Level):
Impact
Based on an evidence
review, what activities
will address these
determinants? What do
you do? How long will
it take?
Intermediate
Objective (Individual
Level):
Instructions: First discuss your target population. Using data, evidence-based recommendations (the Community Guide or
others), your own knowledge, and group discussion, develop a program strategy for controlling diabetes in your
community. Define the goal, objectives, activities, and costs: and describe them in this sample logic model.
16
Learning Objectives
1. Identify key characteristics of successful
action planning
2. Understand the purpose and use of logic
models
3. Describe steps used in constructing logic
model
17
Step1: What is the goal of
your program




What are the health issues facing your
community?
 What is the magnitude of the problem?
Who is most affected, geographic
differences, time trends?
What are the consequences of the problem?
What causes it?
How do you find the answers to these
questions?
18
Step 2: Prioritization
Group process to prioritize which problem(s) to
work on
key components: importance and changeability
priorities
19
Helpful websites
Community Engagement: Community Analysis
and Needs Assessment
See the Community toolbox –
http://ctb.ku.edu/
20
Step 3: Setting specific objectives
a. What are objectives?
 There must be sound scientific evidence to support the
objectives, should be appropriate for community
 The result to be achieved should be important and
understandable to a broad audience
 Objectives should be prevention-oriented
 Objectives should drive action and suggest a set of
interim steps (intermediate indicators)
21
Step 3: Setting specific objectives
a. What are objectives?(cont)
 The language of objectives should be precise, avoiding
use of general or vague verbs
 Objectives should be measurable and may include a range
of measures—health outcomes, behavioral risk factors,
health service indicators, and assessments of community
capacity
 Specific timetables for completion of objectives should be
described
 SMART
22
Step 3: Setting specific objectives
a. What are objectives?(cont)
 Example of a sound objective:
To reduce the percent of women, ages 35 to 55, who are
overweight (BMI>24.9) from 80% in 2000 to 70% in
2010.
 Objectives are achievements, not activities
 The what, not the how
23
Step 3: Setting specific objectives
a. What are objectives?(cont)
 Consider “intermediate” objectives
 if we promise too much in outcomes, we are destined to
fail
 What are intermediate objectives for the BMI example
(what do we need to accomplish to reduce BMI)?
 more on this in evaluation
24
Step 3: Setting specific objectives
b.Consider strategies based on theory
Theories
 explain behavior
 suggest ways to effect change
 guide intervention strategies
 guide development of objectives (e.g.,suggest what
needs to be done to change behavior)
25
Transtheoretical Model
Stages of Change
Individual level theory
 Precontemplation - information
 Contemplation –information, cognitive
restructuring – identity as a smoker/non-smoker
 Preparation- skill building (recipes)
 Action- skill building, environmental cues
 Maintenance- above plus social norms
26
Social Cognitive Theory
 Cognitive processes & behavior
 Social support
 Environment
Create change through- Role modeling/observational learning
 Group meetings
 Reducing exposure to key triggers
27
Strategies at Various Levels
Individual
Interpersonal
Organizational
Objectives
Address
- knowledge
- attitudes
- behavior
- practices
- social support
- social networks
- programs
- practices
- policies
Approaches
- brochures
- training
- counseling
- develop new
social ties
- lay health
advisors
- peer support
groups
- org change
- networking
- development
28
Strategies at Various Levels
(cont)
Community
Policy/Governmental
Objectives
Address
- programs
- environment
- policies
- regulations
- ordinances
- policies
- legislation
Approaches
- infrastructure
- media advocacy
- community
development
- political action
- lobbying
- policy advocacy
29
Step 3: Setting specific objectives
a. What are objectives?(cont)
 Examples of intermediate objectives (promoting
activity) – based on theoretical constructs:
 Knowledge
 Increase the percentage of individuals reporting they understand
the health benefits of physical activity from 40% in 2010 to 50%
in 2015.
 Attitudes/intentions to change
 Increase the percentage of people reporting they intend to
become physically active in the next six months from 12% in
2010 to 22% in 2015.
30
Step 3: Setting specific objectives
c. Consider strategies that others have used and
the local population and context
 Community guide
 Feasibility/Practicality
 Adapt to local environment, policies, cultures,
norms
31
Step 4: Developing Action Strategies
 Be sure to show the link between goals,
objectives & strategies (check using logic
model)
 Developing the evaluation plan (covered
tomorrow)
 Consider specific costs associated with each of
the action steps planned
32
Example
 In order to decrease diabetes (goal)
 We need to increase fruit & vegetable
consumption (objective)
 By increasing knowledge about and access to
fruits and vegetables (objective)
 To increase knowledge we will …(action strategy)
 To increase access we will …(action strategy)
33
Logic Model (Analytic Framework) Worksheet: Evidence-Based Public Health
Program Title:__________________________________________
Goal:
Long-Term Objective:
What are the evidencebased determinants?
Intermediate Objective
(Environmental Level):
Activities:
Activities:
Activities:
Activities:
Costs:
Costs:
Costs:
Costs:
Evaluation
Intermediate
Objective (Govt./Org.
Level):
Process
How much will it cost?
What other resources
are needed
d?
Intermediate
Objective (Social
Level):
Impact
Based on an evidence
review, what activities
will address these
determinants? What do
you do? How long will
it take?
Intermediate
Objective (Individual
Level):
Instructions: First discuss your target population. Using data, evidence-based recommendations (the Community Guide or
others), your own knowledge, and group discussion, develop a program strategy for controlling diabetes in your
community. Define the goal, objectives, activities, and costs: and describe them in this sample logic model.
34
Step 4: Developing Action
Strategies
 Developing the work plan and timetables
 Basic time line construction includes
 A complete listing of activities, grouped by
major categories
 Ascertaining which activities need to be done
first
 Determining how long each activity will take
 Determining when each and every activity is
to begin and finish
 Establishing the time units that are most
appropriate (e.g., weeks, months, years)
35
Step 4: Developing Action
Strategies

Assessing resource needs
1. Available funds: how many direct funds are available? What
are the sources? Are there limitations on how and when
funds can be spent? Are funds internal or external to your
program or agency?
2. Personnel: how many and what types of personnel are
needed? What type of training will be needed for program
staff? Are there “in-kind” funds? What personnel do
collaborating organizations bring to the project?
3. Equipment and materials: what types of equipment and
supplies are needed for the program? Are there certain
pieces of equipment that can be obtained “in-kind” from
participating partners?
36
Step 4: Developing Action Strategies

Assessing resource needs (cont)
4. Facilities: for some types of interventions, significant
infrastructure is needed in the form of clinics, hospitals, or
mobile vans. These need to be carefully considered as they
can amount to a significant expense.
5. Travel: is there travel directly related to carrying out the
project? Are there related travel expenses for other
meetings or presentations in professional settings?
37
Step 4: Developing Action
Strategies

Identifying and training workers
 In which areas does each staff member need
training?
 Who should conduct the training?
 Do some people have unused skills that could be
useful to your program?
 How best should community members be oriented
and trained regarding your program?
 How can training be time efficient?
38
Step 4: Developing Action
Strategies

Reasons for piloting
1. Refine the original hypotheses and/or research
questions
2. Produce information that will help improve evaluation
approaches
3. Improve curriculum materials or evaluation instruments
4. Test approaches for data imputation and analysis
39
Step 4: Developing Action
Strategies

Reasons for piloting (cont)
5. Uncover politically sensitive issues
6. Estimate costs for people, equipment, materials, and
time
7. Ascertain the cultural appropriateness of interventions
in diverse populations by inclusion on program
development
8. Enhance the “marketability” of an intervention with
senior agency administrators when a pilot test is
successful
40
Resources
 Turn to ready-made resources
 Community tool box
 http://ctb.ku.edu/
 Community Health Promotion Kit
 http://www.health.state.mn.us/divs/fh/chp/hpkit/
 Guidelines for Comprehensive Programs to
Promote Healthy Eating and Physical Activity
 http://www.astphnd.org/programs/guidelines.htm
41
Logic Model (Analytic Framework) Worksheet: Evidence-Based Public Health
Program Title:__________________________________________
Goal:
Long-Term Objective:
What are the evidencebased determinants?
Intermediate Objective
(Environmental Level):
Activities:
Activities:
Activities:
Activities:
Costs:
Costs:
Costs:
Costs:
Evaluation
Intermediate
Objective (Govt./Org.
Level):
Process
How much will it cost?
What other resources
are needed
d?
Intermediate
Objective (Social
Level):
Impact
Based on an evidence
review, what activities
will address these
determinants? What do
you do? How long will
it take?
Intermediate
Objective (Individual
Level):
Instructions: First discuss your target population. Using data, evidence-based recommendations (the Community Guide or
others), your own knowledge, and group discussion, develop a program strategy for controlling diabetes in your
community. Define the goal, objectives, activities, and costs: and describe them in this sample logic model.
42