Transcript Monitoring and evaluating capacitybuilding at multiple levels UNFCCC Meeting on monitoring and evaluating CB, Rio, Brazil 6-7 November 2008 David Watson Consultant.

Monitoring and evaluating capacitybuilding at multiple levels
UNFCCC Meeting on monitoring and
evaluating CB, Rio, Brazil
6-7 November 2008
David Watson
Consultant
Scope of Presentation
• Background: perspectives and
approach
• The story so far: M&E of CB under
UNFCCC
• How other global programmes tackle
M&E of CB
• The big picture: M&E of CB in
international development
• Alternative paradigms: esp. systems
thinking
• Towards a tool-kit for practitioners
Background
• Sorry about late distribution of paper! (See
Section X for a potted summary!)
• Personal ‘points of departure’: building
institutions and governance
• A touch of scepticism about M&E practices;
pragmatism; incrementalism; practicality;
process
• Broadening debate constructively
• Not ‘selling’ any particular approaches
• Cognisant of the seriousness of the
challenges of slowing climate change, and of
enhancing the effectiveness of CB to that end
Levels of Capacity and CB
• May be helpful to bear in mind two
complementary dimensions:
• ‘Horizontal’: from individuals, through
organisations, to social / national
systems or networks
• ‘Vertical’ : from local, to regional, to
national, to global.
Other Global Programmes’
Experience with M&E of CB
•
•
•
•
•
•
PRSPs
GAVI / Roll Back Malaria
UNAIDS
Programme Based Approaches (PBAs)
WFP
Lessons pertinent to CB to address
Climate Change
Lessons from other global progs.
• Several (GAVI and RBM) had specialised
groups focussed on M&E, plus Task Forces
on specialised subjects (e.g. CB)
• At country level: a Secretariat may have a
M&E Cell; use existing M&E structures where
they exist; keep it simple; minimise number of
indicators.
• Empirical evidence of enhanced capacities is
often scarce
• ‘Strategic incrementalism’ attracted PBA-ers
attention: pursue ‘quick wins’ and ‘peerlearning’
..lessons continued…
• Clarify objectives of CB, based on thorough
needs assessments, as a basis for a simple
results framework
• Try to avoid a ‘missing middle’: indicators for
the steps between outputs and outcomes;
• Reflect on whether the M&E process should
feed more into national political processes,
(engaging e.g. with parliaments, media, civil
society) AND be a precondition of
partnerships nationally and with the
international community.
• Don’t be naïve in your ‘theory of political
change’: …’participation’ is not enough.
A poignant quotation
‘M&E practices do not provide a
framework for re-presenting (or making
meaning from) the complex and multiple
processes of institutional and individual
learning and change that are sought in
strengthening institutional capacity’
Source: Unitar ‘Challenges and
Constraints in M&E of Capacity
Building’ presentation to first (Antigua)
workshop Nov 2007
•
•
•
•
•
•
•
M&E of CB : recent insights
Theme paper for ECDPM Study on Capacity
Change and Performance 2003-8 (Baser
Morgan + Watson on Theme Paper)
On M&E of CD tended to agree with UNITAR!
Often, ‘performance improvement’ indicators
are used as proxy for capacity increases
…not many e.g.s of ‘capacity’ being
monitored
Accountability (to donors) the main ‘driver’
Most public sector CB experience
disappointing…political and institutional
factors seem important
Donors have a poor record on M&E of CB:
..including their own capacities and incentives
‘Capacitated’ organisations:
1. ..carry out tasks effectively
2. ..engage, decide, act
3. ..relate, attract resources & support
4. ..adapt and self-renew
5. ..balance diversity, and build coherence
across the organisational system
Capacity: ‘that emergent combination of
individual competences and collective
capabilities which enables a human
system to create public value
..and implications for M&E of CD..?
• Agree on the nature of capacity to be
monitored!
• ..need to focus on more complex issues…
• Information on change and progress should
not be sent ‘up’ …but shared internally for
purpose of learning
• Pay more attention to the context…and interrelationships in capacity processes..
• to the changes taking place (intended or not)
and their contributions to ‘capacity’…
• ..i.e. need more participative approaches to
M&E… and to learning and reflection..
Alternative Paradigms to M&E
(of CB)
• Critical reactions to ‘monitoring abilities to
perform…and meeting pre-determined
objectives’
• Wheatley: ‘measurement’ = anathema (when
looking at human behaviour)
• ‘Cause and effect’ logic inapplicable
• ..change processes are complex..involve
inter-dependencies…and are not ‘linear’ or
stable, nor necessarily visible
• ..but may well be long-term, and more
political than technical
•
Systems Thinking’s relevance
• Growth in interest in ST as an analytical
framework for development and natural
resource management
• Explicitly examined as part of ECDPM study:
helps address implications mentioned
• Human institutions seen as complex adaptive
systems: e.g. climate!
• See Box 1 for a summary of basic concepts
• ‘Mess’ = problems unbounded in scope, time
and resources; no clear agreement on
optimal solution or how to achieve it; goals
and strategies are contested; perspectives
differ; no clear cause and effect; uncertainty
prevails.
What about ‘Capacity’?
• ‘Capacity is an emergent property of human
systems’ (ECDPM Study)
• (Emergent properties are those which have
no meaning in terms of parts making up the
whole system. They are the outcome of
system behaviour or synergy.)
• Two examples: ENACT Jamaica (Box 2)
• And IUCN Asia (Box 3)
ENACT
• Formal performance
monitoring system
abandoned
• Empowerment of
frontline staff for rapid
response
• Absence of a ‘model’ to
assess performance
• Let partners adapt and
adopt measures
• Donor modified its
approach..more
‘learning-friendly’ faced
with diversity
IUCN
• Unusually diverse
membership
• Flexibility demonstrated
by funding agencies
• Permitted
experimentation,
innovation and creativity
• Evolution of IUCN
learning processes
• ‘Teaming’ process
• Ownership by
governments + IUCN
credibility / legitimacy
• CB= continuous
process ‘no road map,
only a goal’
Cases’ commonalities c.f. ST
• Identification and recognition of goals
• Emphasis on values to be reflected
• Clarity and awareness of mission amongst
clients too
• Leadership encourages experimentation
• Opportunities for learning from experience:
self-assessment and ‘stories’ of positive
experience or changes or errors
• Flexibility to adapt (thru new skills-building) to
new needs / priorities, thru OJT ‘hands-on’
• Informality of M&E systems: responsive to
needs of clients / network members
• Ability to learn from experience is crucial
BUT ‘Reductionist’ approaches are
still relevant…where:
• It is possible to define required capabilities
unambiguously and specifically
• ..and to assess existing ones = ‘gap’
• Therefore easy to define indicators
• Where stakeholders able and willing to define
their shortfalls and ‘sign up’
• Incentives exist to improve performance
• Leadership, and all above combine into
‘ownership’
• BUT ..this combination of circumstances is
rare! (e.g. public financial management IMF
and WB forged consensus)
Towards a ‘Tool Box’ for M&E of
CD in Climate Change
• Principles: pragmatism: acknowledge
weaknesses in all M&E systems: main
aim = ‘sense-making’
• Seek out ‘what M&E exists, and what
works, already’
• Acknowledge that the best M&E
systems are ‘customised’: adapted by
participants, based on local conditions
Some existing frameworks
• GEF Resource Kit on M&E
• GEF Indicators for 5 key capacities:
– Engagement; generate access to and use of
information; policy and legislation development;
management and implementation; monitor and
evaluate
– Scoring / indicator system for each;
– Obliges actions / next steps / link to outcomes
• UNEP ‘Lessons Trees’
– Trying to improve quality of learning (especially
about common problems) and application of
learning to future programmes
Self-Assessment
• Examples from CB in research and
development organisations
• SA workshops after applying qualitative and
quantitative tools
• Managers, staff and stakeholders identify
strengths and weaknesses, and set new
directions ….Advantages =
(1) those with knowledge of and interest in the
organisation gain in-depth insights: what’s working
and why..where improvements needed
(2) Well-prepared to address the changes needed
• E.g. M&E of past CB in Mekong Farming
R&D Systems Institute (Box 4)
– Preparation of ‘work stories’ on past CB efforts
– + interviews with key staff on changes and
challenges
A ‘Balanced Approach’ to M&E of
CB
• Framework generated to address M&E
of CB a la ECDPM Study dimensions of
‘Capacity’
• Piloted in PNG legal and judicial reform
• Time-consuming and exhaustive: but
national practitioners did find it helpful in
conceptualising all dimensions of
capacities being built up.
• See extra handout
Appreciative Enquiry in Formative
Evaluation
• Evaluators form more of an understanding of
the political, cultural and historical ‘landscape’
• Encouraging organisations to develop their
relationships with primary stakeholders
• Community Development Corporations in US
– Seeking out ‘what enabled effectiveness, and
what hindered it’
– Evaluator regularly listening to ‘stories’
– Regular visits and sharing of reports
– Recipients able to ‘co-create’ the initiative and
develop OWN capacity for assessment
– Evaluator chosen by NGO not donor: NGO
employed not as expert but for its ability to learn
collectively
Most Significant Change (MSC)
• First applied in evaluation of a complex RD
programme in Bangladesh
– Process managers identify domains of change
which are important to evaluate
– Stories (descriptions of changes deemed
significant: with reasons: why significant?)
periodically collected from stakeholders
– Analysed and filtered up thru committees
– Criteria for choosing stories are collated and fed
back to stakeholders;
– Final selection made (annually?) with reasons
– Circulated to all
– Site visits to check; deepen understanding of
changes
MSC features
• Focuses attention and direction of work in
programme towards valued directions
• Dialogue and deliberations crucial
• Takes place over time: responsive to
changing contexts
• Policy makers, funders, field engaged looking
at the value of changes
• Stories help all relate to information
• Non-experts (the story-tellers) involved in
evaluation
• Dialogue based on real experience and
concrete outcomes not abstract indicators
• …MSC positively evaluated as technique in
Laos (Willetts 2004)
Annotated Bibliography
• ..use as a part of the ‘tool-kit’!
• Sections on:
– Climate Change-related sources
– Broader CB literature and M&E
– Other Global Programmes’ Materials
– Civil Society CB and M&E
– Systems Thinking and Complexity
literature and M&E examples
Summary: why ST has potential for
M&E of CB in Climate Change
• Avoids pitfalls of logical framework in what is
an amorphous field: CB
• Emphasises clarity of objective-setting, AND
of learning collectively from reality
• Generates, relates to, and values ‘stories’
• Potential to enhance team-work and interrelationships of hitherto disparate institutions
and groups in their ‘ecosystem’ context;
• Climate change context – politically-charged,
formal and informal, amenable to negotiation
• Evidence indicates ST M&E approaches ‘can
work for and strengthen social change’ (Guijt
2007 IDS)