Transcript Overseas Development Institute

Evaluation and the
Science of Complexity
Evaluating Complexity Conference
NORAD
29th -30th May 2008
Agenda
Evaluations – some common issues
 Complexity science – origins and ideas
 Implications for evaluations
 Summary

The history of M&E in the international
development sector is in four distinct phases

1960s to 1979: Early developments

1979-1984: Rapidly growing interest

1984 to 1988: M&E matures

1988 to the present: the crossroads
Wealth of tools, techniques and
approaches are now available











Logical framework analysis
Results-based management
Needs Assessments, Impact Assessments
Ex ante and ex post assessments
ZOPP
GANTT
Social Network Analysis
Appreciative Enquiry
Most Significant Change
Outcome Mapping
Many many more!!
For many organisations, evaluations are
at the centre of a vicious circle...
Increased
competition
Growing need
for high profile
fundraising
and advocacy work
Poor learning and
accountability
...causing
problems
for M&E
Increased
pressure to show
results and impact
Lack of
professional
norms and
standards
Evaluations are still largely focused on reports as
opposed to changed behaviours, ways of thinking
and attitudes
Reflection, learning and analysis are threatened
by existing agency cultures and processes
Existing
culture &
process
Accountability
Org.
Learning
Evaluation
Evaluations, like other similar initiatives, often sit on top
of existing organisational silos, inefficiencies and power
imbalances, rather than resolving them
Evaluation
Silos
KM
Agencies plough the same evaluation
“field”, but stick to their own furrows
Understanding of the effectiveness
and use of evaluations is weak at
best…
Where’s the
data??!
Evaluations tend to be based on wish
lists, not strategies, and therefore are
often overloaded
Leadership and political buy-in to
evaluation is rare and unreliable, with two
common reactions
...all of which means that (1) evaluation
efforts resemble this iceberg...
What is
planned
What
actually
happens
And (2) evaluators spend most of their time feeling
like this...
working against this...
Agenda
Evaluations – some common issues
 Complexity science – origins and ideas
 Implications for evaluations
 Summary

“Exploring the science
of complexity”






Primary aim was to explore the potential value of
complexity science for those who work on change and
reform initiatives within the aid sector
Drew on scientific and experimental literature –
physiology, physics, mathematics, public sector reform,
sociology, economics, organisational theory, plus case
studies, reports and evaluations from the aid sector
Reviewed over 250 articles, books, reports and
evaluations
10 peer reviewers, including 5 directors of leading
research institutes
Published February 2008
Available to download from www.odi.org.uk
A (well-known) story…
A man was walking home one dark and foggy night. As he
groped his way through the murk he nearly tripped over
someone crawling around by a lamp post.
“What are you doing?” asked the traveler.
“I’m looking for my keys” replied the other.
“Are you sure you lost them here?” asked the traveler.
“I’m not sure at all,” came the reply, “but if I haven’t lost
them near this lamp I don’t stand a chance of finding them.”
A closer inspection of the light
under the lamp revealed…
INPUTS
ACTIVITIES
OUTPUTS
OUTCOMES
IMPACTS
Logical frameworks / results chains articulate a
clear world view and theory of change: “The light
under the lamp”





The machine metaphor - universe
as clockwork
The future is knowable given
enough data points
Phenomena can be reduced to
simple cause & effect
relationships
Dissecting discrete parts would
reveal how the whole system
worked; science was the search
for the search for the basic
building blocks
The role of scientists,
technologists & leaders was to
predict and control - increasing
levels of control (over nature, over
people, over things) would
improve processes, organisations,
quality of life, entire human
societies
Key Assumptions

Assumptions about systems



Assumptions about how systems change




Ordered
Reductionist - parts would reveal the whole
Linear relationships
influence as direct result of force from one object to another predictable
Simple cause & effect
Assumptions about human actions



Rational choice
Behavior specified from top down
Certainty and “knowability”
...Reality of aid is a little different...
But a new light is being turned on (slowly,
unevenly, using a dimmer switch)…
Complexity science is a science of
understanding change

A loosely bound collection of ideas, principles and
influences from a number of other bodies of knowledge,
including







chaos theory
fractal geometry
cybernetics
complex adaptive systems
postmodernism
systems thinking
Discovery of similar patterns, processes and
relationships in a wide variety of phenomena

related to the nature and dynamics of change
From microscopic chemical reactions…
...to the evolution of galaxies...
Complexity scientists use a range of
ideas and concepts (familiar,
challenging and baffling) to make
distinctions between simple,
complicated and complex systems
and phenomena
Simple
Following a Recipe






The recipe is essential
Recipes are tested to
assure replicability of
later efforts
No particular
expertise; knowing
how to cook
increases success
Recipe notes the
quantity and nature of
“parts” needed
Recipes produce
standard products
Certainty of same
results every time
Complicated
Complex
A Rocket to the Moon
Raising a Child

Formulae are critical
and necessary

Formulae have only a
limited application

Sending one rocket
increases assurance
that next will be ok

Raising one child
gives no assurance of
success with the next

High level of expertise
in many specialized
fields + coordination

Expertise can help but
is not sufficient;
relationships are key

Separate into parts
and then coordinate

Can’t separate parts
from the whole

Rockets similar in
critical ways

Every child is unique

Uncertainty of
outcome remains

High degree of
certainty of outcome
The claims of complexity scientists






The complexity of real world systems is (usually) not
recognised or acknowledged by scientists and policy
makers
Or, that if it is not acknowledged, they don’t deal with
them
Or, that if they do deal with them, they don’t do so
effectively
Or, that if they do deal with them effectively, it’s because
they used an specific approach / framework
...that is also available to you, dear client, at a
reasonable daily rate plus a per diem
[JOKE]
There have been diverse efforts to
apply ideas to social, economic and
political analysis and practice










Arthur, Ormerod - Economics
Stacey, Snowden - Organisations
Jervis, Urry, Cutler - Intl relations
De Mancha - History
Gilchrist - Community development
Education policy - Sanders and McCabe
Health policy - Zimmerman
Government reform - Chapman
Strategic thinking – Saunders
Evaluation - Williams
Complexity and systems approaches
have already proved useful in rethinking aid and development issues












Uphoff, 1990s
Chambers, 1997
Sellamna, 1999
IDRC, Outcome Mapping, 2001
Warner, 2001
Rihani, 2002
Lansing and Miller, 2003
Inclusive Aid, 2004
ECDPM, 2004-06
Eyben, 2006
Guijt, various
Davies, Network Analysis, various
10 key concepts and implications
Features of systems
1 Interconnected
and interdependent
elements and dimensions
3 Emergence
from
Simple
Rules
2 Feedback
processes
Dynamics of change
4 Non-Linearity
5 Sensitivity
to initial
conditions
7
Strange
attractors and
the edge of
chaos
6 Phase
space and
attractors
Behaviours and relationships
8 Adaptive
Agents
9 Self
`
organisation
10 Co-Evolution
Complex systems
Collection of parts, which collectively
have a range of dimensions
 Parts share an physical or symbolic
environment / space
 Action by any part can affect the whole

 E.g.
individuals, families, communities, cities,
markets, societies, populations, economies,
nations, planets
Complex systems are interconnected and
interdependent to different degrees

Interconnectedness may occur between any elements,
dimensions, systems and environments

This interconnectedness leads to interdependence between the
elements and the dimensions of a system, and gives rise to complex
behaviour

Complex systems can be tightly or loosely coupled, internally
and with their environment, giving rise to different kinds of
complex behaviours



Tightly coupled: Global FOREX markets
Loosely coupled: US University system, global construction industry
Systems can be understood via mapping techniques, followed by
analysis to understand the dynamics and interactions of the
system
Foot and mouth disease: an example of failure
caused by focusing on one part of the system and
ignoring the links between sub-systems (biology,
geography, economics)
Economic rationalisation of abattoirs
and EU subsidies increased the
interconnectedness of herds to a
critical point.
Changes to foot and mouth reporting
rules delayed the isolation of
infectious animal
The relationship between these
actions and the epidemiology of F&M
was not appreciated in advance
where it mattered because the
livestock industry was not viewed
as a interconnected,
interdependent system
Complexity also means that systems need to be
understood at different scales
Communities
Atom
Organisms
Molecule
Tissue
Cell
Organs
E.g. evaluating the effectiveness of
child health programmes
Private
Sector
Other NGOs
A.N. NGO
Local
partners
Developing
Country
Govmts
Community
and
Family
Church
Civil Society
Example: evaluating resource flows in the
humanitarian system
Issues of interconnectedness,
interdependence and scale usually do not
become apparent until a crisis...

Foot and mouth disease, UK


September 11th


Biofuels and food consumption
Vulnerability to natural disasters


US mortgage market mis-selling and the world economy
Food prices


Western consumerism and Southern disasters
Credit crunch


Globalisation and terrorism
Climate change


Economics, cattle management, disease
Sichuan earthquakes and dams
Human trafficking

Desire, economics and rights abuses
...indicating that we have biases in the way
we view the world...


Three different kinds of problems have been
identified, along with some common biases in
dealing with them
They are:
 “Messes”
 “Problems”
 “Puzzles”
“Messes” are issues that do not have
a well defined form or structure. NB
not a value statement, but a
description

There is often not a clear understanding of the problem faced

Messes often involve economic, technological, ethical and political
issues


It has been suggested that all of the really important issues in the world
start out as messes.
For example, how was rising HIV/AIDS incidence in Brazil dealt with
in the 1990s?

concerned money, technology, ethics, social relations, politics, gender
relations, poverty
 all of these dimensions of the problem had to be dealt with
simultaneously, and as a whole
Many of the major problems we face are
“messes”!

Credit crunch


Food prices


Sexual preferences and human rights abuses
Arms trade


Globalisation and terrorism
Human trafficking


Western consumerism and Southern disasters
September 11th


Sichuan earthquakes and dams
Climate change


Biofuels and
Vulnerability to natural disasters


US mortgage market mis-selling and the world economy
Economics and war
Etc, etc, etc
“Problems” are issues that have a known or
knowable form or structure

The key dimensions and variables of a problem are known and the
interaction of dimensions may also be understood, even if only
partially.

With problems, there is no single clear cut way of doing things



there are many alternative solutions, depending on the constraints faced
Expertise matters
For example, dealing with the sewage system in a particular city
may rely on amount of money available, technology, political stance
of leaders, climatic conditions, urban development, the road system,
and so on
Puzzles are well defined and well
structured known problems with a
specific “best” solution

Solutions can be worked out and improved

Solutions are replicable - “best practices” are possible
Policy and traditional science is
biased towards puzzle-solving

Real-world, complex, messy nature of
systems is frequently not recognised

Simple puzzle-based solutions are applied
to complex messes
 E.g.
Global War on Terror has been applied
as the single best solution to the mess of
terrorism
“Some of the greatest mistakes have been made
when dealing with a mess, by not seeing its
dimensions in their entirety, carving off a part,
and dealing with this part as if it were a problem,
and then solving it as if it were a puzzle, all the
while ignoring the linkages and connections to
other dimensions of the mess”
Or to put it another way: dividing a
cow in half does not give you
two smaller cows
Implications: analyse and deal with
the reality of the system

Multidimensionality, interdependence and interconnectedness of
poverty and humanitarian crises (and responses to them) should be
recognised by those designing, managing and evaluating aid interventions

Analysis may need to be in line with historical research - not ‘did x
cause y?’ but ‘what happened and why?’, building narratives about
events and processes.

The task of selection and synthesis of data becomes as important as
analysis

Different perspectives on what the system is need to be taken into
account, especially when these perspectives differ as to the nature,
interconnectedness and scale of the system

Messes, problems and puzzles need to be identified and dealt with
using appropriate approaches
Interconnectedness and interdependence
gives rise to a range of phenomena and
behaviours
To find out more, read the paper!
Features of systems
1 Interconnected
and interdependent
elements and dimensions
2 Feedback
processes
Dynamics of change
4 Non-Linearity
5 Sensitivity
to initial
conditions
6 Phase
space and
attractors
3 Emergence
from
Simple
Rules
7
Strange
attractors and
the edge of
chaos
Behaviour of agents
8 Adaptive
Agents
9 Self
`
organisation
10 Co-Evolution
Agenda
Evaluations – some common issues
 Complexity science – origins and ideas
 Implications for evaluations
 Summary

Evaluation of Complex Systems is
NOT new

Educational systems, social initiatives and
government interventions are complex social
systems where effective evaluation is seen as a
key process in measuring success

But there is an increasing recognition that for
evaluation of complex social systems to be more
effective, evaluations may need to take into
account the theoretical understanding of
complex systems
Implications for evaluation are at
four levels
1.
2.
3.
4.
Implications for evaluation methodologies
and approaches
Implications for the focus of evaluations
Implications for the purpose and scope of
evaluations
Implications for evaluations as a complex
system in their own right
Implications for evaluation
methodologies and approaches
Linear models dominate...
Certain
resources
are needed
to operate
your program
Inputs
If you
If you have
accomplish
If you
access to
your planned
deliver the
them, then
activities,
product and /
you can use then hopefully or service to
the extent
them to
you will deliver
achieve your the amount of Intended, then
planned
the product your participants
will benefit
activities
and / or
in certain ways
service that
you intended
Activities
MONITORING
Outputs
Outcomes /
Purpose
EVALUATION
If these
benefits to
participants
are achieved,
then certain
changes in
communities,
organisations
and systems
might be
expected
to occur
Impacts /
Goal
Many evaluation “results chains” visualize
change as linear, based on multiple cause-effect
logic models
Y
X
Inputs
Activities
Outputs
Outcomes
Linear, Predictable
Focused on the end result
The program (X) gets the credit!
Impact
When applied to development, “the
results chain” can lead to
Faulty thinking
 Misguided data collection
 Misleading reporting of results
 Gives a false sense of reality to senior
managers and donors who are far from
where change is taking place

Complexity science sees change as…
 Interconnected (multiple actors and factors)
 Non-linear (unexpected results occur)
 Incremental, cumulative, with tipping points
 Beyond the control of the project / programme
 Two-way (program also changes)
 Continuous (not limited to the life of the project)
Therefore
Develop new theories of change, adapt
existing theories of change to challenge
assumptions of linearity
 Design evaluations to allow for
interdependencies and interconnections in
the system the program is trying to
influence, and capture the resulting
dynamics

Implications for evaluation focus
Focus of “traditional” evaluations

Formal project / programme / organisation

Environment is outside the organisation


evolves separately until programme is implemented
Level of analysis is single or at most a few, relatively
independent levels
Implications of complexity



Features of systems: evaluate from perspective of
multiple, nested levels of interconnected systems, study
feedback between the organisation and its environment,
look for emergent rather than planned change
Dynamics and nature of change: Look for non-linearity,
anticipate surprises and unexpected outcomes, analyse
the system dynamics over time and frame the “space for
possible change”, look for changes in conditions that
facilitate systemic change, and how well matched the
programme is to the wider system
People, motivations and relationships: Study patterns of
incentives and interactions among agents, study quality
of relationships, study individuals and informal / shadow
coalitions, vs. formal organisation, study co-evolution of
organisation and environment
Implications for evaluation
purpose and scope
M&E is seen as standing in
contrast to creative dynamism of
field work
Purpose and scope of traditional
evaluations vs complexity-oriented
evaluations
Traditional
Complexity-oriented
Measure success against
predetermined goals
Develop new measures and
monitoring mechanisms as goals
emerge & evolve
Render definitive judgments of
success or failure
Provide feedback, generate
learning, support direction or affirm
changes in direction
Aim to produce generalisable
findings across time & space
Aim to produce context-specific
understandings that inform ongoing
innovation
Creates fear of failure
Supports hunger for learning
Implications for evaluation as a
complex system
Key Assumptions

Assumptions about systems



Assumptions about how systems change




Ordered
Reductionist - parts would reveal the whole
Linear relationships
influence as direct result of force from one object to another predictable
Simple cause & effect
Assumptions about human actions



Rational choice
Behavior specified from top down
Certainty and “knowability”
Evaluations traditionally seen as a rational,
technical, information-generating process
...Reality is a little different...
Goals
Activities
Monitoring
Evaluation
Results
Evaluation systems are themselves complex
systems, with many interconnected parts and
dimensions; no two evaluations are the same












Focus of evaluation
Policy / Guidelines
Scope Project vs Policy
Demand, goals
Timing, quantity
Preparation, TOR
Management and team selection
Methods and tools
Engagement with stakeholders
Dissemination of findings and utilisation
Costs
Quality maintenance mechanisms
“...To be effective an evaluation program
must match the dynamics of the system to
which it is applied....”
Eoyang and Berkas (1998)
Implications of evaluations as a
complex system
Traditional
Complexity-oriented
Position the evaluator outside to
assure independence and
objectivity
Position evaluation as an internal,
team function integrated into action
and ongoing interpretive processes
Accountability to control and locate Learning to respond to lack of
blame for failures
control and stay in touch with
what’s unfolding and thereby
respond strategically
Accountability focused on and
Accountability centred on
directed to external authorities and fundamental values and
funders
commitments
Evaluator controls the evaluation
and determines the design based
on the evaluator’s perspective
about what is important
Evaluator collaborates in the
change effort to design a process
that matches philosophically and
organisationally
All systems can be placed on a
spectrum between extremes of
ordered and chaotic

E.g. solids and gases

In solids, atoms are locked into place
 In gases they tumble over one another at random

In between the two extremes, at a phase transition, a phenomenon
called the ‘edge of chaos’ occurs


This phenomenon describes systems behaviours where the evolution
of the system never quite locks into place and never quite dissolve into
turbulence either.
In human organisations, the simplest example is of a system that is
neither too centrally controlled (order) nor too bottom-up and
therefore disorganised (chaos)
Physiology on the Edge of Chaos
Severe Congestive
Heart Failure –
orderly
Healthy Heart –
on the edge of
chaos
Cardiac Arrhythmia,
Atrial Fibrillation chaotic
Dynamic adaptability is the key
to system health
Point of
Maximum
Adaptability
High
Adaptability
ZONE of HEALTH
Changelessness is a sign of death,
transformation a sign of life.
- Commentary on the I Ching
Low
Threshold
Ordered
Threshold
Dynamics
Disordered
Evaluation at the
edge of chaos?
Agenda
Evaluations – some common issues
 Complexity science – origins and ideas
 Implications for evaluations
 Final points

Complexity science gives additional
weight to calls for re-thinking






The wider contexts of aid work
The nature of the processes involved in aid work
The dynamics of change involved in aid work
The real influence of aid work
The role of partner organisations and
beneficiaries in aid work
The tools and techniques for planning,
monitoring, learning and evaluating aid work
Given the resistance to change in the power
dynamics of aid, real world applications may
continue to be “innovative”, “under the radar”
and outside the mainstream of aid policy
and practice...
There are a number of common
criticisms of complexity...
Theoretical: adds nothing new



E.g. the ideas of complexity science have all been identified elsewhere
But complexity brings them together
Practical: doesn’t specify what should be done



E.g makes no specific recommendations as to how best to act in complex
systems
See next slide
Supports managerial “snake-oil” / “complexologists” / re-warmed ideas



E.g. is abused and misused
What isn’t?
Political: emergence and self organisation support neo-liberal stances



Just because self-organisation happens, doesn’t mean it is good

Credit crisis

Rwandan genocide
Complexity is more centre-ground, for example, “edge of chaos” systems are
seen as most robust and resilient because are the optimally combination of
control and flexibility
...and different perspectives on the
value of complexity science

Deep paradigmatic insights
 Champions

Interesting parallels and useful approaches, but
not the only way to see things
 Pragmatists

Meaningless coincidences
 Critics
Source of weakness is also the
source of strength

These ideas are not about “what you dos”, but about
“how you dos”



Not “solutions for problems”, but “approaches to problems”
Tools for furthering understanding, for opening up new ways of
seeing and thinking
They point to the personal, professional, institutional,
political mindsets, attitudes and conditions which need to
be in place to work effectively in and with complex
systems
Complexity concepts can support
the intuition and navigation of
practitioners
INPUTS
ACTIVITIES
OUTPUTS
OUTCOMES
IMPACTS
Four suggestions

Develop collective intellectual openness to ask a new, potentially valuable,
but challenging set of questions of our mission and their work

Develop collective intellectual and methodological restraint to accept the
limitations of a new and potentially valuable set of ideas

not misuse or abuse or let them become part of the ever-swinging pendulum of
aid approaches

Need to be humble and honest about the scope of what can be achieved
through ‘outsider’ interventions, about the kinds of mistakes that are so
often made, and about the reasons why such mistakes are repeated

Need to develop the individual, institutional and political courage to face up
to the implications of complexity
Final points (1)
...We can't solve
problems by using the
same kind of thinking
we used when we
created them..
Final points (2)
Everybody thinks
to change the
world;
nobody thinks to
change
themselves
Thank you!

Get in touch
 [email protected]