presented at Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention Research Center University of North Carolina at Chapel.
Download
Report
Transcript presented at Evaluation Considerations: Measures & Methods Shrikant I. Bangdiwala, PhD Professor of Research in Biostatistics Injury Prevention Research Center University of North Carolina at Chapel.
presented at
Evaluation Considerations:
Measures & Methods
Shrikant I. Bangdiwala, PhD
Professor of Research in Biostatistics
Injury Prevention Research Center
University of North Carolina at Chapel Hill, USA
1
WHO VIP Webinar 2011
Content
Purpose of evaluation
Cycle of program planning & evaluation
Indicators
Study designs
Statistical modeling
Challenges
2
WHO VIP Webinar 2011
What are we ‘evaluating’?
Actions, programs, activities
Conducted in a community setting, over a
period of time
Aimed at reducing deaths, injuries, and/or
events and behaviors that cause injuries
3
WHO
Safety
VIP2010
Webinar
London
2011
Example:
Suwon,
South
Korea
area of
‘safety
promotion’
http://www.phs.ki.se/csp/safecom/suwon2.htm
4
WHO VIP Webinar 2011
Why do we ‘evaluate’?
To know ourselves what works and if we are doing
some good
In performing some activity
In the community
In the country
To convince funders and supporters that their
investment is worthwhile
To convince the community about the benefits of the
multiple activities and actions carried out
5
WHO VIP Webinar 2011
Main purposes of evaluation
Evaluation helps determine:
How well a program/policy works relative to its
goals & objectives
Why a program/policy did or didn’t work, relative
to planned process
How to restructure a program/policy to make it
work, or work better
Whether to change funding for a program
6
WHOChicago
VIP Webinar
NSC
20102011
6
Methodological complications
Multiplicities
Multiple components of a program
Multiple populations at risk
Multiple study designs
Multiple types of effects/impacts/outcomes &
severities
Multiple audiences/objectives of ‘evaluation’
Multiple methods for conducting evaluation
7
WHO VIP Webinar 2011
When should evaluation be
considered?
Evaluation needs to begin in, and be part of,
the planning process…
Otherwise, “if you do not know where you are
going, it does not matter which way you go,
and you will never know if you got there or
not!”
Lewis Carroll (1872)
Alice in Wonderland
8
Adapted from M. Garrettson
WHO VIP Webinar 2011
8
Types of evaluation depending
on program phase
Program
Planning
Phase
Program
Implementation
Phase
Post
Program
Phase
9
Formative Evaluation
How can the program activities be improved before
implementation?
Process Evaluation
How is/was the program (being) implemented?
Impact / Outcome
Did the program succeed in achieving the intended
impact or outcome?
WHOChicago
VIP Webinar
NSC
20102011
9
Cycle of program planning and
evaluation
10
Adapted from C Runyan
WHO VIP Webinar 2011
Identify population & problem
Surveillance data
Other needs assessment strategies
key informant interviews
focus groups
surveys
evaluations of past programs
literature
consultation with peers
other info…
11
WHO VIP Webinar 2011
11
Disseminate
Identify
problem
& population
Evaluation:
•Process
•Impact
•Outcome
Identify
resources
Implement
Test, Refine,
Implement
Test & refine
implementation
12
Define target
audience
Evaluation:
•Formative
Choose
strategies
WHOChicago
VIP Webinar
NSC
20102011
Set goals/
objectives
12
Define target audience
To whom is the program directed?
Whose injuries need to be reduced?
Who is the target of the program?
•
•
•
•
•
13
at risk persons
care givers (e.g. parents)
general public
media
decision makers
WHO VIP Webinar 2011
13
Understand target audience
What are their characteristics?
Special needs (e.g. literacy)
Interests, concerns, priorities
Attitudes & beliefs re: problem & solutions to
problem
Cultural issues
14
WHO VIP Webinar 2011
14
Disseminate
Identify
problem
& population
Evaluation:
•Process
•Impact
•Outcome
Identify
resources
Implement
Test, Refine,
Implement
Test & refine
implementation
15
Define target
audience
Evaluation:
•Formative
Choose
strategies
WHO VIP Webinar 2011
Set goals/
objectives
15
Identify resources
Community partners
interest in topic
working on similar projects
On-going activities
Sources of financial support
Interests in community
16
WHO VIP Webinar 2011
16
Disseminate
Identify
problem
& population
Evaluation:
•Process
•Impact
•Outcome
Identify
resources
Implement
Test, Refine,
Implement
Test & refine
implementation
17
Define target
audience
Evaluation:
•Formative
Choose
strategies
WHO VIP Webinar 2011
Set goals/
objectives
17
Set goals & objectives
Goal
broad statement of what program is trying to
accomplish
Objectives
Specific
Measurable
Time-framed
18
WHO VIP Webinar 2011
18
Disseminate
Identify
problem
& population
Evaluation:
•Process
•Impact
•Outcome
Identify
resources
Implement
Test, Refine,
Implement
Test & refine
implementation
19
Define target
audience
Evaluation:
•Formative
Choose
strategies
WHO VIP Webinar 2011
Set goals/
objectives
19
Choose Strategies
Identify existing strategies/programs
Literature: evidence based? promising practice?
WHO manuals
Successes from other communities-regionscountries
Develop new strategies:
Logic model (how would it work)
Haddon matrix
20
WHO VIP Webinar 2011
20
Haddon Matrix
Person
Vehicle/
vector
Physical
Environ.
Social
Environ.
Preevent
Event
Postevent
21
Haddon 1970 Am J Public Health
WHO VIP Webinar 2011
21
3-dimensional Haddon Matrix
Other??
Feasibility
Preferences
Stigmatization
Equity
Freedom
Cost
Event
Post-event
Effectiveness
22
Runyan 1998 Injury Prevention
WHO VIP Webinar 2011
22
Disseminate
Identify
problem
& population
Evaluation:
•Process
•Impact
•Outcome
Identify
resources
Implement
Test, Refine,
Implement
Test & refine
implementation
23
Define target
audience
Evaluation:
•Formative
Choose
strategies
WHOChicago
VIP Webinar
NSC
20102011
Set goals/
objectives
23
Formative Evaluation
Why it’s useful
Questions it answers
What is the best way to
influence the target
population?
Will the activities reach the
people intended, be
understood and accepted
by target population?
How can activities be
improved?
24
Improves (pilot-tests)
program activities before
full-scale implementation
May increase likelihood
program or policy will
succeed
May help stretch resources
WHO VIP Webinar 2011
24
* Modified from Thompson & McClintock, 2000
Disseminate
Identify
problem
& population
Evaluation:
•Process
•Impact
•Outcome
Identify
resources
Implement
Test, Refine,
Implement
Test & refine
implementation
25
Define target
audience
Evaluation:
•Formative
Choose
strategies
WHOChicago
VIP Webinar
NSC
20102011
Set goals/
objectives
25
Implementation
As planned, with attention to detail
Documented clearly so others can replicate if
appropriate
26
WHO VIP Webinar 2011
26
Disseminate
Identify
problem
& population
Evaluation:
•Process
•Impact
•Outcome
Identify
resources
Implement
Test, Refine,
Implement
Test & refine
implementation
27
Define target
audience
Evaluation:
•Formative
Choose
strategies
WHOChicago
VIP Webinar
NSC
20102011
Set goals/
objectives
27
Process evaluation
Purpose is to address:
What was done?
How was it implemented?
How well was it implemented?
Was it implemented as planned?
28
WHO VIP Webinar 2011
28
Process evaluation –
examples of questions
•
•
•
•
Who carried out intervention?
Was this the appropriate person/group?
Who supported and opposed intervention?
What methods/activities were used?
29
WHO VIP Webinar 2011
29
Process evaluation
- why is it useful?
•
•
30
Allows replication of programs that work.
Helps understand why programs fail.
WHO VIP Webinar 2011
30
* Modified from Thompson & McClintock, 2000
The intervention cannot be a
black box…
It must be clearly understood
Idea
31
?
WHO VIP Webinar 2011
Outcome
31
Disseminate
Identify
problem
& population
Evaluation:
•Process
•Impact
•Outcome
Identify
resources
Implement
Test, Refine,
Implement
Test & refine
implementation
32
Define target
audience
Evaluation:
•Formative
Choose
strategies
WHOChicago
VIP Webinar
NSC
20102011
Set goals/
objectives
32
Impact evaluation
Purpose is to address changes in:
33
knowledge
attitudes
beliefs/ values
skills
behaviors / practices
WHO VIP Webinar 2011
33
Using impact measures for
Establishing effectiveness
Suppose we have a public safety campaign as our
strategy
Need to show
Campaign
Outcome
If we already have demonstrated that
Behavior
Behavior
Outcome
We simply need to show
Campaign
34
Behavior
WHO VIP Webinar 2011
34
Outcome evaluation
Purpose is to address changes in:
injury events (e.g. frequency, type, pattern)
morbidity (e.g. frequency, severity, type)
mortality (e.g. frequency, time to death)
cost (e.g. direct and indirect)
35
WHO VIP Webinar 2011
35
Example: Bike helmets
Intervention
Physician
counseling
parents
Enforcement of
helmet law
Impacts
Outcomes
Parental
attitudes
toward child
helmet use
Head injury in
bike crashes
Purchase of
helmets
Deaths from head
injury in crashes
Media campaign
Use of helmets
by children
36
WHOChicago
VIP Webinar
NSC
20102011
36
Evaluation examples of questions
for local policy of smoke alarms
Did the local policy of smoke alarms in apartments…
Get passed
Where people aware of it?
Did people have access to smoke alarms?
Did people get them installed properly?
Do people keep them maintained?
Lead to a reduction in the number or rates of:
events (e.g. apartment fires)
injuries
deaths
37
costs (e.g. burn centerWHOcosts,
family
burden, property loss)
VIP Webinar
2011
37
Evaluation – selection of
measures
‘Quantitative Indicators’
Process
Impact
Outcome
Health related
Financial
38
WHO VIP Webinar 2011
38
Choice of measure or indicator
We need to choose appropriate impact and outcome measures
‘Soft’ (more difficult to measure) outcomes –
Perceptions constructs: fear, insecurity, wellbeing, quality of life
Knowledge, attitudes and behaviors constructs
Hard outcomes –
Deaths, hospitalizations, disabilities due to injuries and violence
Societal impacts – local development indicators
Economics outcomes –
Direct $/€/£/¥, indirect DALYs, QALYs, opportunities lost, burdens
39
WHO VIP Webinar 2011
Evidence of effectiveness
Obtain qualitative ‘evidence’ to complement
the quantitative ‘evidence’
Ex. Are “multisectorial collaborations and
partnerships” friendly and functioning well?
Ex. Is “community participation” optimal?
Incorporate process indicators
Incorporate narratives & testimonials
40
WHO VIP Webinar 2011
Disseminate
Identify
problem
& population
Evaluation:
•Process
•Impact
•Outcome
Identify
resources
Implement
Test, Refine,
Implement
Test & refine
implementation
41
Define target
audience
Evaluation:
•Formative
Choose
strategies
WHOChicago
VIP Webinar
NSC
20102011
Set goals/
objectives
41
Dissemination
Dissemination not done well
Not attempted
Not based on research about how to disseminate
information to intended audience
Dissemination done well
Defining audience
How to access audience
How best to communicate change message to them
Presentation of clear, straightforward messages
42
WHO VIP Webinar 2011
42
Evaluation measures
Lead to evidence of effectiveness
But only if the research and study
methodologies, and the statistical analyses
methodologies, are appropriate to convince
the funders and supporters, the skeptics,
the stakeholders, the community
43
and understandable
WHO VIP Webinar 2011
43
Research methodology approach:
Evidence of effectiveness
Obtain quantitative ‘evidence’ that favors the hypothesis that
the intervention is effective as opposed to the (null) hypothesis
that the intervention is not effective.
How?
Experimental study designs - randomized clinical trials,
grouped randomized experiments, community-randomized
studies
Quasi-experimental study designs - non-randomized
comparative studies, before-after studies
Observational studies - cohort studies, case-control studies
and comparative cross-sectional studies
44
WHO VIP Webinar 2011
Randomized controlled trial
(RCT) / Experiment
‘strongest’ evidence
Intervention
Group
O
X
O
O
X’
O
Randomize
Control
Group
45
WHO VIP Webinar 2011
45
Quasi-experimental designs
‘qualified’ evidence
Intervention Group
Comparison Group
46
O
O
X
O
O
WHO VIP Webinar 2011
46
One group pre/post
‘weak’ evidence
Intervention
Group
47
O
X
O
WHO VIP Webinar 2011
47
One group –
multiple pre / multiple post
better ‘weak’ evidence
Intervention
Group
48
O
O
O
X
O
O
O
O
O
WHO VIP Webinar 2011
48
One group, post only
‘basically ignorable’ evidence
Intervention
Group
49
X
O
WHO VIP Webinar 2011
49
Observational designs
- cohort study
evidence?
Self-chosen Intervention Group
Self-chosen Non-intervention Group
50
X
X
X
O
O
O
WHO
Safety
VIP2010
Webinar
London
2011
50
Observational designs
- case-control study
evidence?
X
Cases
O
X
Controls
O
51
WHO
Safety
VIP2010
Webinar
London
2011
51
Observational designs
- cross-sectional study
evidence?
X
X
O Injured X
O
O
X
Non-injured O
O
O
52
WHO
Safety
VIP2010
Webinar
London
2011
52
Statistical analysis
methodologies
Choice - often guided by what has been done
previously, or what is feasible to do, or easy to
explain
Choice should be tailored to the audience &
their ability to understand results; but also on
the ability of the presenter to explain the
methodologies
53
WHO VIP Webinar 2011
Statistical analysis
Determined by research question(s)
Guided by study design – experimental or observational
Guided by whether outcome is studied at a single time point or
multiple time points
Group randomized controlled experiment
Non-randomized comparison study
Single site pre/post; surveillance study
Retrospective or cross-sectional
Time series analyses
Guided by audience
VIP Webinar 2011
Visual
and descriptiveWHO
appreciation
54
Visual and descriptive analysis
– longitudinal time series
Example:
Espitia et al (2008)
Salud Pública Mexico
55
WHO VIP Webinar 2011
Visual and descriptive analysis
– comparisons over time
Example:
www.gapminder.org
56
WHO
Safety
VIP2010
Webinar
London
2011
Statistical analysis - challenge
But what we as a field have not done as well as other
fields, is to draw strength from numbers develop
collective evidence
Combine results from multiple studies
Systematic reviews (of observational studies)
Meta analysis (of experimental & observational studies)
Meta regression (of heterogeneous studies)
Mixed treatment meta regression (for indirect
comparisons)
57
WHO VIP Webinar 2011
Systematic reviews
A protocol driven comprehensive review and
synthesis of data focusing on a topic or on related key
questions
58
formulate specific key questions
developing a protocol
refining the questions of interest
conducting a literature search for evidence
selecting studies that meet the inclusion criteria
appraising the studies critically
synthesizing and interpreting the results
WHO VIP Webinar 2011
Example –
Systematic review
Shults et al (2001) Amer J Prev Med
59
WHO VIP Webinar 2011
Systematic reviews
Of particular value in bringing together a number
of separately conducted studies, sometimes with
conflicting findings, and synthesizing their results.
Zaza et al (2001) Amer. J Preventive Medicine – motor vehicle
To this end, systematic reviews may or may not
include a statistical synthesis called meta-analysis,
depending on whether the studies are similar
enough so that combining their results is
Green (2005) Singapore Medical Journal
meaningful
60
WHO VIP Webinar 2011
Meta analysis
A method of combining the results of studies
quantitatively to obtain a summary estimate of the
effect of an intervention
61
Often restricted to randomized controlled trials
Recently, the Cochrane Collaboration is
‘branching out’ to include both experimental and
observational studies in meta analyses
WHO VIP Webinar 2011
Meta analysis
e.g. Liu et al
(2008)
Cochrane
Collaboration
62
WHO VIP Webinar 2011
Meta analysis
The combining of results should take into account:
the ‘quality’ of the studies
• Assessed by the reciprocal of the variance
the ‘heterogeneity’ among the studies
• Assessed by the variance between studies
63
WHO VIP Webinar 2011
Meta analysis
– estimation of effect
The estimate is a weighted average, where the weight of a study
is the reciprocal of its variance
In order to calculate the variance of a study, one can use either a
‘fixed’ effects model or a ‘mixed’/’random’ effects model
Fixed effects model:
utilizes no information from other studies var(Yi ) var(ei ) VYi
Random effects model:
considers variance among and within studies Yi i ei
var( ) 2
var(Yi ) 2 VY*i
WHO VIP Webinar 2011
64
Borenstein et al (2009) Introduction to Meta Analysis
Meta analysis & meta regression
Dealing with ‘heterogeneity’ among the studies - 2
Decompose the total variance into among and within
components using mixed effects models for
getting a more precise estimate of the intervention
effect
If there is still residual heterogeneity
65
Expand the mixed effects model to include study-level
covariates that may explain some of the residual
variability among studies meta regression
WHO VIP Webinar 2011
Meta regression
e.g.
Yi 1 X1i 2 X 2i i ei
random error
study random effect
Overall mean
X1 study variable – EU/USA
X2 study variable – population type
66
WHO VIP Webinar 2011
Meta analysis
Standard meta-analytical methods are typically restricted
to comparisons of 2 interventions using direct, head-tohead evidence alone.
So, for example, if we are interested in the Intervention A
vs Intervention B comparison, we would include only
studies that compare Intervention A versus Intervention B
directly.
Many times we have multiple types of interventions for
the same type of problem, and we hardly have head-tohead comparisons
We may also have multiple component interventions
67
WHO VIP Webinar 2011
Mixed treatment meta analysis
Let the outcome variable be a binary response
1 = positive response
0 = negative response
We can calculate the binomial counts rj:k out of a total
number at risk n j:k on the kth intervention in the jth study
rj:k
We can then calculate
p
j:k
n j:k
the estimated probability of the outcome (risk of response)
for the kth intervention in the jth study
68
Welton et al 2009 Amer J Epid
WHO VIP Webinar 2011
Mixed treatment meta analysis
Let each study have a reference ‘‘standard’’ intervention arm,
sj, with study-specific ‘‘standard’’ log odds of outcome, j .
The log odds ratio, j:k, of outcome for intervention k, relative
to standard sj, is assumed to come from a random effects
model with
mean log odds ratio (dk ds j ) , and
between-study standard deviation
where dk is the mean log odds ratio of outcome for
intervention k relative to control (so that d1 = 0).
69
Welton et al 2009 Amer J Epid
WHO VIP Webinar 2011
Mixed treatment meta analysis
This leads to the following logistic regression model:
ln
where
70
Welton et al 2009 Amer J Epid
p j:k
1 p j:k
j
j j:k
j:k ~ N[(dk ds ), ]
j
WHO VIP Webinar 2011
int s j
int k
Mixed treatment meta analysis
- multiple-methods interventions
If we have multiple methods in the ith intervention
M1i , M 2i , M 3i ,...
Plus we have multiple times when the outcome is
assessed
Yit i 1t 2 M1it 3M 2it 4 X i eit
Study effect
Components 1 & 2 effects
Time effect
71
WHO VIP Webinar 2011
Study covariable
Error term
Statistical analysis
Methodology does exist for developing stronger
collective evidence, evaluating the effectiveness of
community based interventions, using different types
of study designs and interventions
Developing “practice-based evidence”
72
WHO VIP Webinar 2011
Dissemination
We should not stop at developing the evidence
We must work alongside economists in developing
ways to effectively communicate ‘what works’
methodology and cost models do exist for estimating
the “return on investment”
Money talks !!
73
WHO
Safety
VIP2010
Webinar
London
2011
Challenges – Evaluation requires
Integration of evaluation from the beginning
Appropriate measures, possible to be collected objectively,
unbiasedly, easily and with completeness
Appropriate qualitative and process information, to complement
the quantitative information
Concrete and convincing evidence of what aspects work in
individual communities
Formal methodological statistical evaluation of specific elements
of programs
Collective evidence of what common elements of programs work
Effective dissemination strategies – “return on investment”
74
WHO VIP Webinar 2011
75
WHO VIP Webinar 2011
75