Evaluation of Influence
• Why measure impact?
• What to measure?
• What sort of policy change?
• Planning/measuring in complex environments
• Behaviour change
• Some tools we use
• Other tools
• A systematic approach
Why should you do M&E?
get more funds
keep funders/clients happy
• (“what is the guarantee that doing this actually
What should you measure?
“If you don't know
where you are
going, any road
will get you there”
Policy processes are...
RAPID Outcome Mapping Approach
• Discursive: Clientfocused services
• Attitudinal: Farmers
have good ideas
• Procedural: Participatory
approaches to service development
• Content: UU20, UU25. New guidelines
• Behavioural: Approach being applied in
Visibility or substance?
Short term ‘relevant’ research
Long term research
Focus on solutions for ‘agreed’
Engage with the definition of the problem
Lobby, network, horse trading
Briefing papers, Opinion pieces
Estimates, costed proposals, policy options
Website, Blogs, Facebook, etc.
Academic publications, long reports
Online communities with millions of
Communities with the right people
Delegations at high level global
Private meetings at Party conferences and
private meetings while planning for the high
Event focused influence
Problem focused influence
Prospect magazine Think Tank of the year
The Cynefn Framework
Focus on behaviour change
Professionalisation of Public
Structural Adjustment → collapse
Paravet projects emerge.
projects – collaborative
ITDG Paravet network
network.and change of DVS.
Dr KajumeRapid spread in North. The Hubl
KVB letter (January 1998).
Multistakeholder WSs → new policies.
Still not approved / passed!
Sarah Earl, Fred
Carden, and Terry
Social Network Analysis
RAPID Outcome Assessment
After Action Reviews
• What was supposed to
• What actually happened?
• Why was there a difference?
• What can we learn from this?
15 minute team
in a “rank-free”
Most Significant Change
1. Collect “stories of
change” from different
2. Systematic analysis of
ODI Impact Logs
Contact: [email protected]
• Classical case studies (IDRC, IFPRI)
• Stories of Change (Denning)
• Innovation Histories (CIAT)
• HERG Payback Framework (Hanney)
• Micro-Narratives (Snowden)
• Impact matrices (Davies)
• Peer evaluations (CHSRF)
• Systematic reviews?
A systematic approach
1. Evidence and advice:
• Peer-review of outputs
• Uptake logs
• Outcome mapping
2. Public campaigns and advocacy
• Surveys and focus groups
• Media tracking logs
• Media/public frame analysis
• Records of meetings
• Tracking people
• Key informants
A systematic approach
1. Strategy and direction –are you
doing the right thing?
2. Management –are you doing what
you planned to do?
3. Outputs – are the outputs
appropriate for the audience?
4. Uptake – are people aware of your
5. Outcomes and impacts –are you
having any impact?
3ie PIM Project – Initial ideas
Research to influence:
• Clear objectives
• Understand the context
• Theory of change
• Iterative / learning
Theory of change
Expect the unexpected
• Strategy: theory of change, impact
pathway, peer review, log frame
• Management: appreciative inquiry, logs,
• Output: logs, peer-review,
• Uptake: logs, webstats, surveys
• Impact: outcome mapping, stories of
change, episode studies, peer review
Helping researchers become policy entrepreneurs. How to develop
engagement strategies for evidence-based policy-making. John
Young and Enrique Mendizabal. ODI Briefing Paper 53. 2009 http://www.odi.org.uk/resources/details.asp?id=1127&title=becom
Outcome Mapping: Building Learning and Reflection into
Development Programs. Sarah Earl, Fred Carden, and Terry
Smutylo IDRC - http://www.idrc.ca/EN/Resources/Publications
Making a difference: M&E of policy research - ODI Working Paper
281. Ingie Hovland. July 2007. http://www.odi.org.ukwww.odi.org.
A guide to monitoring and evaluating policy influence. Harry JOnes.
ODI Background Notes, February 2011 - http://www.odi.org.uk/