Transcript Slide 1

Monitoring and
Evaluation of Influence
Enrique Mendizabal
June 2011
Outline
• The skeptical view
• Why monitor and evaluate?
• What can we monitor and evaluate?
• Types of influencing
• Methods and tools for each
• A suggestion
• Work on your own
The skeptics
3
(on think tanks) some experts say:
From Braml (2004)
• Kent Weaver: it is hard to determine policy influence of one
think tank in relation to others because they are all
different, work on different aspects of a policy, influence
in different ways and there are many other players
involved.
• Nelson Poslky: there is no sense in looking for direct
influence of think tanks’ activities because one can
only ask these questions when one ignores the
complexity of political processes. Some causal
relationships may only be found in a few cases, but
systematic explanations of this sort remain an illusion.
more
Weidenbaum (2009)
• Andrew Rich: dollar for dollar, think tanks attract much
more attention than any other organisation.
• John Hamre, of the Center for Strategic and International
Studies: evaluating the influence of think tanks on a
particular policy would be like determining who is the
father of Anna Nicole Smith’s baby. As soon as the
change happens every think tank who had something to
say about it is quick to claim it for itself.
• David Frum, ofthe American Enterprise Institute: since one
cannot measure influence, [no method] would do it.
5
even more
Abelson (2006)
• Abelson: by focusing on the influence of think tanks on
policy change one would overlook a great deal of other
types of influence that is the source of think tanks’
value.
• Leslie Gelb, form the New York Times: influence is highly
episodic, arbitrary and therefore difficult to predict.
• Abelson and Ricci: as the policy community becomes more
open and complex it also becomes more difficult to say
anything meaningful about the contribution of think
tanks to it.
6
and more
Puryear (1994)
• The most interesting and valuable contributions of
think tanks [in Chile] may not be intellectual but
psychological: the hundreds of events and seminars
that Chilean think tanks organised throughout the
1980s helped to restore the mutual trust and
understanding missing from Chilean politics and that
had led to the rupture of democratic order.
7
And:
Hoppe (2010)
The search for ‘impact’ is driven by three
assumptions:
• Unidirectional transfer of advice;
• Policymakers and experts performing different
and exclusive tasks;
• All possible impact is desirable per se
8
In practice, however:
• Robust research into uptake and impact will be so
costly and time consuming, that nobody would
be able to afford it.
• Lessons drawn will be marginal and irrelevant
for other situations.
• Indicators focusing on that can be measured
would only lead to conformance and perverse
conduct.
9
Why monitor and
evaluate?
10
Why should you do M&E?
•
•
•
•
To
To
To
To
learn
manage better
get more funds
keep funders/clients happy
• (“what is the guarantee that doing this actually helps
us?”)
Researchers vs research
Do we want:
1) Our research/IE to be more influential?
or
2) Policy to be based on evidence?
Visibility or substance?
Visibility
Substance
Short term ‘relevant’ research
Long term research
Focus on solutions for ‘agreed’
problems
Engage with the definition of the problem
Media exposure
Lobby, network, horse trading
Briefing papers, Opinion pieces
Estimates, costed proposals, policy options
Website, Blogs, Facebook, etc.
Academic publications, long reports
Online communities with millions of
hits
Communities with the right people
Delegations at high level global
conferences
Private meetings at Party conferences and
private meetings while planning for the high
level conferences
Event focused influence
Problem focused influence
Global Go-To-Survey
Prospect magazine Think Tank of the year
What are we
evaluating? –and
some complications
14
M&E & Learning
1.
Strategy and
1. Strategy and direction –are you doing the right
things?
2. Management –are you doing what you planned
to do?
3. Outputs – are outputs up to scratch?
4. Uptake – are people aware of your work?
5. Outcomes and impacts –are you having any
impact?
How?
Hovland (2009):
1.
Strategy and direction: Logframes; Social Network Analysis; Impact
Pathways; Modular Matrices
2.
Management: ‘Fit for Purpose’ Reviews; ‘Lighter Touch’ Quality Audits;
Horizontal Evaluation; Appreciative Inquiry
3.
Outputs: Evaluating academic articles and research reports; Evaluating
policy and briefing papers; Evaluating websites; Evaluating networks;
After Action Reviews
4.
Uptake: Impact Logs; New Areas for Citation Analysis; User Surveys
5.
Outcomes and impacts: Outcome Mapping; RAPID Outcome
Assessment; Most Significant Change; Innovation Histories; Episode
Studies
16
M&E of policy influence – what’s
the problem?
Influencing
Activities
POLICY
And who does the influencing?
• The researchers
• Campaigners/third parties?
• Policymakers themselves?
Losing control
Project
Team
Other
Actors
Inputs
Activities
Outputs
Outcomes
Outcomes
Outcome
s
Policy
changes
Impact
Impact
Impact
So we need to look for proxys
“Government runs in part, on the basis of memos. If a
SD or DD official, or an analyst at the CIEA or the NSC,
has your study in front of him and open at the time he is
writing his own memo to the secretary or the director or
perhaps the president himself –if, in short, he is using
your ideas and analysis at the time he writes his own
memo- then you have influence. If your study is not
open in front of him, or worse, you do not even know
who the responsible official is, you do not have
influence. It is as simple as that.”
Howard Wiarda quoted by Abelson D. E., 2006
24
Types of influencing
What to measure and how
3 types of influencing approaches
(there are more):
• Evidence and advice
• Public debate/education and
advocacy
• Lobbying approaches
Methods and tools
Evidence and advice
How?
What to measure
Tools
Outputs
Evaluating research
reports, policy briefs
and websites
Consultancy
Research and
analysis, ‘good
practice’
Evidence-based
argument
Logs;
Uptake and use
user surveys
RAPID outcome
assessment;
Providing advisory
support
Developing and
piloting new policy
approaches
new areas for citation
analysis;
Influence
Episode studies;
Most Significant
Change
Public debate/education and advocacy
How?
What to measure
Tools
Surveys,
Public
Target audience
communications and attitudes, behaviour,
campaigns
etc
focus groups,
direct responses,
career paths
University teaching
Media tracking logs,
‘Public education’
Media attention
media assessment
Debate
Messaging
Advocacy
Media framing and
influence
Framing analysis;
coverage
Lobbying approaches
How?
Face-to-face meetings
and discussions
Relationships and
trust
Direct incentives and
diplomacy
What to measure
Tools
Actors;
Recording
meetings;
relationships;
tracking people;
policy process and
institutions
interviewing key
informants;
probing influence
Recommendations
• Develop a Theory of Change based on sound
theory
• Integrate PME for policy influence
• Use a mix of tools to gather data
• Ask: What does it mean?
A suggestion
An example (based on ODI’s structure)
33
Develop your own
Exercise
• Describe the change you would like to see
• Describe the role you will play in it -your contribution
to this change
• How will you know that you have fulfilled this role to
the best of your abilities?
• How will you know whether you are doing the right
thing?
• How will you incorporate lessons (that you and others
have learned) into your planning and implementation?
Additional resources
www.odi.org.uk/rapid
www.ebpdn.org
www.outcomemapping.ca
www.onthinktanks.org