Transcript Slide 1

Monitoring and
Evaluation of Influence
John Young
Outline
• Why measure impact?
• What to measure?
• What sort of policy change?
• Planning/measuring in complex environments
• Behaviour change
• Some tools we use
• Other tools
• A systematic approach
• Recommendations
• Resources
2
Why should you do M&E?
•
•
•
•
To
To
To
To
learn
manage better
get more funds
keep funders/clients happy
• (“what is the guarantee that doing this actually
helps us?”)
3
What should you measure?
“If you don't know
where you are
going, any road
will get you there”
4
Policy processes are...
Cabinet
Donors
Policy
Formulation
Agenda
Setting
Parliament
Decision
Making
Civil Society
Monitoring and
Evaluation
5
Private
Sector
Ministries
Policy
Implementation
RAPID Outcome Mapping Approach
6
Policy objectives
• Discursive: Clientfocused services
• Attitudinal: Farmers
have good ideas
• Procedural: Participatory
approaches to service development
• Content: UU20, UU25. New guidelines
• Behavioural: Approach being applied in
practice
7
Visibility or substance?
Visibility
Substance
Short term ‘relevant’ research
Long term research
Focus on solutions for ‘agreed’
problems
Engage with the definition of the problem
Media exposure
Lobby, network, horse trading
Briefing papers, Opinion pieces
Estimates, costed proposals, policy options
Website, Blogs, Facebook, etc.
Academic publications, long reports
Online communities with millions of
hits
Communities with the right people
Delegations at high level global
conferences
Private meetings at Party conferences and
private meetings while planning for the high
level conferences
Event focused influence
Problem focused influence
Global Go-To-Survey
Prospect magazine Think Tank of the year
8
The Cynefn Framework
9
Focus on behaviour change
Project
Team
Other
Actors
Inputs
10
Activities
Outputs
Outcomes
Outcomes
Outcome
Behaviour
Change
Impact
Impact
Impact
Episode Studies
1970s
Professionalisation of Public
Services.
International
Research
of services.
Structural Adjustment → collapse
1980s
Paravet projects emerge.
projects – collaborative
ITDG projects.
research.
Privatisation
Privatisation.
ITDG Paravet network
network.and change of DVS.
1990s
Dr KajumeRapid spread in North. The Hubl
Study
KVB letter (January 1998).
Multistakeholder WSs → new policies.
2000s
Still not approved / passed!
11
Outcome Mapping
OUTCOME
MAPPING:
Building
Learning and
Reflection into
Development
Programs
Sarah Earl, Fred
Carden, and Terry
Smutylo
http://www.idrc.ca/en/ev-9330-201-1DO_TOPIC.html
12
Social Network Analysis
13
www.odi.org.uk/RAPID/Tools/Toolkits/KM/Social_network_analysi
RAPID Outcome Assessment
14
www.odi.org.uk/RAPID/Publications/RAPID_WP_266.htm
After Action Reviews
• What was supposed to
happen?
• What actually happened?
• Why was there a difference?
• What can we learn from this?
15
15 minute team
debrief, conducted
in a “rank-free”
environment.
Most Significant Change
1. Collect “stories of
change” from different
stakeholders
2. Systematic analysis of
“significance”.
16
http://www.mande.co.uk/docs/MSCGuide.pdf
Impact Logs
17
ODI Impact Logs
18
Contact: [email protected]
Others....
• Classical case studies (IDRC, IFPRI)
• Stories of Change (Denning)
• Innovation Histories (CIAT)
• HERG Payback Framework (Hanney)
• Micro-Narratives (Snowden)
• Impact matrices (Davies)
• Peer evaluations (CHSRF)
• Systematic reviews?
• RCTs?
19
A systematic approach
1. Evidence and advice:
• Peer-review of outputs
• Uptake logs
• Outcome mapping
2. Public campaigns and advocacy
• Surveys and focus groups
• Media tracking logs
• Media/public frame analysis
3. Lobbying
• Records of meetings
• Tracking people
• Key informants
20
http://www.odi.org.uk/resources/details.asp?id=5252&title=monitoringevaluation-me-policy-influence
A systematic approach
1. Strategy and direction –are you
doing the right thing?
2. Management –are you doing what
you planned to do?
3. Outputs – are the outputs
appropriate for the audience?
4. Uptake – are people aware of your
work?
5. Outcomes and impacts –are you
having any impact?
21
3ie PIM Project – Initial ideas
22
Conclusions
Research to influence:
Measuring impact
• Clear objectives
•
•
•
•
•
•
• Understand the context
• Theory of change
• Iterative / learning
approach
23
Clear objectives
Theory of change
5-levels
Multiple methods
Triangulation
Expect the unexpected
Recommendations
• Strategy: theory of change, impact
pathway, peer review, log frame
• Management: appreciative inquiry, logs,
AARs, PRINCE2
• Output: logs, peer-review,
• Uptake: logs, webstats, surveys
• Impact: outcome mapping, stories of
change, episode studies, peer review
24
Resources
•
Helping researchers become policy entrepreneurs. How to develop
engagement strategies for evidence-based policy-making. John
Young and Enrique Mendizabal. ODI Briefing Paper 53. 2009 http://www.odi.org.uk/resources/details.asp?id=1127&title=becom
e-policy-entrepreneur-roma
•
Outcome Mapping: Building Learning and Reflection into
Development Programs. Sarah Earl, Fred Carden, and Terry
Smutylo IDRC - http://www.idrc.ca/EN/Resources/Publications
/Pages/IDRCBookDetails.aspx?PublicationID=121
•
Making a difference: M&E of policy research - ODI Working Paper
281. Ingie Hovland. July 2007. http://www.odi.org.ukwww.odi.org.
uk/resources/docs/2426.pdf
•
A guide to monitoring and evaluating policy influence. Harry JOnes.
ODI Background Notes, February 2011 - http://www.odi.org.uk/
resources/details.asp?id=5252&title=monitoring-evaluation-mepolicy-influence
25