Transcript Evaluation and Dissemination of Evidence

The following slides were presented at a
meeting of potential editors and methods
advisors for the proposed Cochrane
review group in February 2008. The slides
were designed to promote discussion
rather than represent the views and
directions of this group.
CONTEXT & PROCESS
Meeting of the Cochrane Public Health Editorial Group
Penny Hawe, Professor, AHFMR Health Scientist &
Markin Chair in Health and Society
Population Health Intervention Research Centre
ww.ucalgary.ca/phirc
PREMISE REGARDING CONTEXT

Effectiveness of some interventions will depend on the interaction
between the intervention and the context into which it is
implemented …
… context is an effect modifier

Context includes the social, economic, political, and organisational
characteristics in host setting

Need to disentangle program effects from (program x context)
interaction effects to discern ‘under what circumstances’
PREMISE REGARDING PROCESS
FAILURE TO SHOW AN EFFECT

Evaluation failure


failed to show an effect when one was there
Programme failure


Inadequacy of intervention theory
Poor implementation (fidelity, intensity, consistency)

Contextual factors

Process measures need to look beyond programme reach (messages
delivered and received) to consider programme stability, intensity, and
consistency with theory
MEASUREMENT OF CONTEXT

Importance of context is recognised in health promotion /
health education literature especially in sustainability /
institutionalisation literature … but mainly qualitative

Largely unrecognised in RCTs … even in cluster RCTs,
which in theory could wash out context level effects

Recent acknowledgement of importance of measuring
and reporting context effects to improve external validity
of evidence (call for improved reporting standards)
CONTEXTUAL FACTORS (HPPH Field)

Availability, accessibility of health-promoting / health harming factors (price
of fruit, number of liquor outlets in a neighbourhood, worksites that ban
smoking)

Aspects of host organisation (staff numbers, skills, morale, competing
priorities, history of innovation, status of programs within organisational
hierarchy, resources made available, what programs are displaced)

Aspects of system (reward structures, discretion, autonomy)

Characteristics of population (literacy, prevalence of problem)

Characteristics of social or organisational networks (size, reach, density,
centrality of key agencies) into which the intervention is placed
EXTERNAL VALIDITY (Green & Glasgow)

OUTCOMES FOR DECISION MAKING






Significance to decision makers
Adverse consequences
Moderator effects including sub-group analysis
Sensitivity: dose-response, threshold effects
Costs: economic vs accounting
MAINTENANCE AND INSTITUTIONALISATION

Long term outcomes 12 months post intervention
 Institutionalisation: sustainability of intervention post evaluation
 Attrition
EXTERNAL VALIDITY (Green & Glasgow)

REACH AND REPRESENTATIVENESS





Participation rate
Target audience stated
Representativeness - settings to target settings and to refusers
Representativeness - individuals to target population and refusers
PROGRAM OR POLICY IMPLEMENTATION

Consistent implementation
 Staff expertise
 Program adaptation
 Mechanisms (processes and mediating variables)
• Oral and written narratives from field workers
• Key informant interviews
• Event or impact logs
• Organisational network analysis before and after
• Additional resources provided to project
Modo Pergite Natare