Transcript Slide 1
IUHPE Global Programme: The
Public Health Agency of
Canada’s Effectiveness Project
Marcia Hills, RN, PhD
Simon Carroll (PhD Student)
Canadian Consortium for Health
Promotion Research
Sylvie Desjardins, Director, Strategic
Policy, PHAC.
Rio de Janeiro, Brazil May 12, 2005
Overview
1) Typical strategies for assessing
effectiveness…systematic reviews and an
alternative approach
2) Policy-maker’s need for evidence of
effectiveness and frameworks/tools
3) The Public Health Agency of Canada’s
Effectiveness Project
Why should health promoters care
about ‘evidence’ and ‘effectiveness’
Increasing demand on policy/decision makers to provide
transparent and explicit justification of decisions
Ignoring the resource allocation issue is neither possible
nor desirable
Valuable lessons are discounted
Valuable evidence is ignored
What counts as evidence
“Gambit of compliance”
Leads to poor evaluations, poor data gathering, and a
waste of valuable resources (i.e. 10% of funds!)
Evaluation…
Programmatic
Focuses on goals, objectives and outcomes of a
particular program
Highly contextualized
Evaluation – programs, people & practice
Concerns practice with people in communities
– What is the relationship between the evaluator
and those delivering the program?
– What should be accepted as evidence upon
which to base practice?
Effectiveness…
Looks for an evidence across programs
Driven by desire for evidence-based
policy…practice…decision-making
Accountability for public funds
Decides what constitutes ‘evidence’ and
influences how evaluations are conducted
Typical Strategies for Assessing
Effectiveness…Systematic Reviews
Numerical Meta- analysis
– Randomized Control Trials
– Quantitative
– Statistical notion of causality (probability)
Narrative Review
– Multiple Methodologies (interpretative)
– Qualitative
– Ignores or rejects causal explanation
Different philosophies, assumptions
Paradigm “wars”
Difficulties with both approaches…
Based on “best buy”
Accumulation of evidence in similar
programs
Beliefs about causality that are based on
statistics or reactions to it
Either context–free or over
contextualized lessons
Realist synthesis approach (Pawson, 2000)
Based on philosophy of science that argues that it is
possible to have a logic of comparison that is not statistical
Causal powers of an intervention (program) lie in its
underlying mechanism/s - its basic theory about how
program resources will affect participant’s action in certain
contexts
NOT …does the program work ? …BUT what are the
conditions (context) under which the resources the program
offers have an impact on participant’s actions
Mechanisms, Contexts,
Outcomes
Want to discover those contexts that have
produced solid and successful outcomes
FROM…those contexts that induce failure
All cases are important …successful policy
depends as much on avoiding previous errors
as by “imitating” successes
Causality…
The language of causality is crucial because it refers to the
elements of a programme that make them work…or not work.
Improving the priority setting of funders and improving the
quality of practice in health promotion demands attention to
what causes positive and negative outcomes
We need to move beyond the either/or of relying on probabilistic
reasoning or retreating into the defensive position of denying
the capability of causal assertions
We need to conceptualize our phenomena into discernable
structures, objects and relations, rather than a obsessive focus
on producing data for statistical modeling.
Public Health Agency of Canada:
Effectiveness Project
Goal..
To develop a framework for assessing the
effectiveness of federally-funded community
initiatives to improve health
Framework for Assessing the Effectiveness of Community
Initiatives That Promote Health
Collaborative
Contexts Planning
Social
Community
Change
Economic
Political
Environment
Gender
Individual, Community
Individual
Transformational
Change & Systems
Change
Change
Systems
Change
Cultural
Community
Organization
& Action
Impact on
Determinants of
Health
Improvements in
Population
Health Outcomes While
Reducing
Health Inequity
Component 1 Mechanisms:
Collaborative Planning
1) Meaningful participation of all relevant
stakeholders
2) Critical dialogue
3) Shared power
4) Project action planning and evaluation
Component 2: Community
Organization and Action
1) Ongoing education and training opportunities
2) Evolving leadership
3) Sustained mobilization of resources
4) Critical reflection and systematic monitoring
Component 4: Transformational
Change
1) Develop and attract champions
2) Generate publicity of project successes
3) Influence Public Policy and Decision-making
Bodies
4) Work with relevant social movements and
provincial and/or national advocacy groups
A Realist Synthesis Approach to
Assessing Effectiveness…
Allows us to learn from past successes and failures
Focuses our attention on the issues so important to
community practitioners
Uncovers what is often undervalued and discounted
Builds credibility for community initiatives by using a
theory of causation based on a realist philosophy of
science
A Snapshot of the NARO Effectiveness Website
URL <http://dynosite.romulin.com/web/uofvic>
Until one is committed, there is hesitancy, the chance to draw
back, always ineffectiveness, concerning all acts of initiative (and
creation). There is one elementary truth, the ignorance of which
kills countless ideas and splendid plans: that the moment that one
definitely commits oneself, then Providence moves too. All sorts of
things occur in one’s favour that would never otherwise have
occurred. A whole stream of events issues from the decision,
raising in one’s favour all manner of unforeseen incidents and
meetings and material assistance which no man could have
dreamed would come his way. Whatever you can do or dream you
can, begin it. Boldness has genius, power and magic in it. Begin it
now!
Johann Wolfgang Von Goethe