Transcript Kampala Evaluation Talk

Uganda Evaluation Association & Evaluation Capacity
Development Project (ECD)
KAMPALA EVALUATION TALK
Kampala, Uganda
22nd Jan., 2015
Inside the presentation (asking questions)
1. Context and Background
(The Case of the Nile Basin Initiative)
2. Typical Operational set op of Regional
Development Programs
3. Key Evaluation Design and Operation Issues?
4. Working within existing situations to achieve
developmental goals
Context and Background
Regional Programs:
 Offer greater benefits, efficiency
& effectiveness in channeling
and receiving development
assistance.
… But:
 Take time to set up & implementation pace is different in each
country
 Require flexible program designs for responsiveness and relevance.
 Participating countries have varied landscapes and Risks (Political,
Policy, Financial, Regulatory).
 Inconsistencies between regional & national programs - through
overlaps, contradictions, differential relevance and applicability.
Regional Development Programs will
typically have 3 tier operational levels
Regional level
with an Executive
Organ
• Coordinates, makes polices &
plans,& commissions evaluations
National level suboffice for
coordinating incountry
implementation:
• Receives policies, work plans,
implementation guidelines and “adapts”
to country context for implementation.
• All participating countries treated as
equals in one whole
Community level
(development design
and evaluation
subjects)
• Ultimately impacted by the development
program will be organized to facilitate receipt
of program support as designed at regional
level
• Program perception and expectations varied
and divergent from the objectives of the
regional program
Nile Basin Initiative (NBI) is a Regional 10
countries’ River Basin Organization in Africa
ATP = Applied Training Project
ATP
NTEAP = Nile Trans-boundary
Egypt
Environment Action Project
CBSI = Confidence Building &
Stakeholder Involvement
NTEAP
SVPC = Shared Vision
Eritrea
Sudan
Coordination Project
Nile-SEC
CBSI
SDBS = Socio-Economic
WRPM
S. Sudan
SVPC
Development & Benefit Sharing
Ethiopia ENTRO
SDBS
EWUAP = Efficient Water Use
Kenya
Project
EWUAP
D.R. Congo
Uganda
RPT = Regional Power Trade
NELCU Rwanda
WRPM = Water Resources
RPT
Planning and Management
Burundi
Tanzania
5
NBI STRATEGIC PROGRAMMING 1999-2016
Cooperation & IWRM
Basin Cooperation and
IWRM building blocks
Dialogue, knowledge & planning
Subsidiary
Action Program
Subsidiary Action
Program
Expanding Nile Basin
Investment Projects
Action on the ground
1999
NBI Sub-basin & Countries
2008
2012
Establishment and
Confidence
Building
Institutional
Strengthening
Consolidation
and
Delivering
NELSPAP &
ENSAP
Shared Vision
Program
Nile-SEC
NBI Secretariat Services
2016 +
NBI - Shared Vision
“To achieve sustainable socio-economic development through the equitable
utilization of, and benefit from, the common Nile basin water resources.”
Strategic Objectives
(2012-2016)
NBI Portfolio of
Investment Projects 20122016
NBI Programs 2012 - 2016
Facilitating Basin
Cooperation
Program
Platform for
Cooperation
Water Resources
Management
Program
Water Resources
Development Program
Equatorial
Lakes
Eastern
Nile
Basin Monitoring
Power
Knowledge base
River basin
management
Analysis
Secretariat
Water Policies
Agriculture
Power
River basin
management
Agriculture
Climate
change
Climate change
Environment Mgnt.
Lead: NILE - SEC
for basin-wide
Lead: NILE - SEC
For basin-wide
Institutional
development
Lead: SAPs
Lead:
Member
States
Approaches, designs, practices and methods for
evaluating regional programs should recognize
complexity & country variations …
 To achieve greater responsiveness of Transboundary/ Regional Evaluations to different
contexts, policies, politics, local aspirations and
communities’ ways of doing things”
Regional Block Countries have variations in:
1.
2.
3.
4.
5.
5.
6.
Levels of rigor in application & use of M&E;
Systems & procedures (Policies, Guidelines, Institutional
Arrangements) approaches & adequacy;
Local capacities, skill levels & expectations at all three
key levels of operation;
Political appreciation and championed support for M&E
versus being perceived as “threat” to the status quo;
Openness to criticism – (threats to conclusion validity,
statistical power analysis, peer review etc);
Ability of local communities to demand accountability;
Times / paces at which costs are incurred and benefits
accrued at different levels (complexity of results chain).
Achieving goals within existing situations
requires multiple approaches to evaluation
designs & implementation.
 Regionally designed & nationally implemented programs to recognize
and read at the same page with national systems and policies – to build
national capacities and increase the efficacy of national systems, & build
on synergies and support national governments achieve their
development goals .
 Mixed open ended evaluation approaches, designs and methods that
measure essentially contribution of the development intervention need
to be applied at different levels of the interventions at different times,
and with staggered levels of rigor - in accordance with program stage of
implementation in a given country.
 Consider different stakeholder perspectives, knowledge, skills and
capacities, governance issues,
Achieving goals within existing situations
requires multiple approaches to evaluation
designs & implementation.
 Encourage local partnerships in addition to deliberate local
capacity building,
 Addressing data constraints - (reconstructing baseline data &
control groups, working with non-equivalent control groups,
collecting data on sensitive topics & difficult to reach groups,
multi-method approaches)
 Need for multiple solutions - outcome mapping and
contribution analysis adapted to multi-country evaluations,
with flexibility to take care of socio-economic, cultural, and
political peculiarities and differences in countries' levels of
"maturity", can greatly increase the relevance and objectivity
of present day evaluation theories.
Achieving goals within existing situations
requires multiple approaches to evaluation
designs & implementation.
 Emphasizing process analysis, to include; one-whole analysis
of complex characteristics of the evaluands, contexts at
different geographic levels.
 Methodologies that seek alternative approaches to the
counterfactual since external events and influence of other
programs are critical in assessing effect of a regional program
on national processes and outcomes.
 Helping clients use the evaluation – (ensuring active
participation of clients in the scoping phase, formative
evaluation strategies, constant communication with all
stakeholders, evaluation capacity building, appropriate
strategies for communicating findings, developing and
monitoring the follow-up action plans)
Useful Questions to ask
whose policies and strategies are
applied when a regional evaluation
is undertaken in a country with
operational national M&E policies
and strategies,?
Would it be possible to design

multiple country-focused localized
evaluations that seek to fulfill ToRs
for evaluation of one regional
development program?
Can an aggregate generalization on
outcomes or impact of a regional

program be made by summing
findings from different targeted

communities obtained through
undifferentiated assessment?
In the sample case of NBI, are any
water resources management changes
and peoples' poverty reduction changes
observed at communities and national
level as a result of NBI or as a result of
own efforts of the National
Governments with some level of
contribution by NBI's? What is this
contribution?
What are the perceptions of the local
communities in each of the countries'
Nile Basin areas?
Can these communities be identified
with the evaluation results at Regional
level?
Concluding with Careful Attribution of Regional
Interventions to National Efforts – The Missing Casual Link
Contribution analysis; assesses the contribution
of a particular development agency (and the
particular intervention supported) to the
achievement of the overall changes resulting from
the collaborative financial and technical
interventions of a number of different
development agencies.
THANKYOU FOR
YOUR TIME
Attribution analysis; is carried out when the
treatment / project group is matched to a comparison /
control group so that alternative explanations of the
observed changes can be controlled for elimination of
other “causes” of changes manifested. There is then a
prima facie case for claiming that the difference
between the treatment / project group and the
comparison / control group indicates the contribution
of the intervention / project to these changes.