Transcript SELECTED SLIDES FOR M&E GROUP

SELECTED SLIDES FOR
M&E GROUP
Evaluation is part of the
Management Cycle
Evaluation
Planning
Implementation
2
Complementary Functions
• Without monitoring, evaluation cannot be done well.
• Monitoring is necessary, but not sufficient for
evaluation.
• Monitoring facilitates evaluation; evaluation generates
additional data collection and information.
• An evaluation of an intervention will often lead to
improvement of plans.
• An evaluability exercise often results in modifying data
collection for monitoring purposes.
3
Evaluation is answering
• Whether we are Doing the Right Things
– Rationale/justification
– Client satisfaction
• Whether we are Doing it Right
– Effectiveness/coherence
– Efficiency: optimizing resources
• Whether there are Better Ways of Doing it
– Alternatives
– Good practices
– Lessons learned
4
Monitoring
Definition
• A process of tracking or measuring
what is happening.
It includes:
• Performance monitoring measuring progress in relation to
an implementation plan;
• Situation monitoring - measuring
change in a condition or a set of
conditions or lack thereof (e.g.,
changes in the situation of women
or changes in the broader socioeconomic country context).
Purpose
• Measure progress according to
plan.
• Trigger timely adaptation of
response, particularly in crises or
unstable contexts;
• Generate critical information for
the evaluation;
• Provide accountability for
implementation;
5
Evaluation
Definition
• Assess the contribution and
worth of an intervention.
• Judgment is based on criteria
such as relevance, efficiency,
effectiveness, impact and
sustainability.
• Evaluation may be conducted
during an intervention (at end
of phase or mid-point) and/or
at the end of the intervention
(ex post).
• Generate lessons learned.
Purpose
• Supports decision-making
• Improve programme
performance;
• Learning by doing;
• Provide accountability for
programme results.
6
What is a result?
Sustainable
A result is a measurable or describable change
arising from a cause and effect relationship.
7
Two major elements:
1. Change
Measurable/ describable transformation
•
•
•
•
Of rights holders’ status
Of beliefs, behaviours and practices
In institutional performance
In quality & availability of Services and other
development tools (products) – knowledge, skills
A result is NOT the completion of activities!!
8
2. Causality
Cause and effect relationship between an action and the
results achieved. “If-then” logic.
Strategic Result
Programme Result
Result
Result
9
Hierarchy of Results
Desired changes in:
• Rights holders’ status - Impact
• Beliefs, behaviour and practices – Outcome
• Institutional performance – Outcome
• Quality & availability of Services and other
development tools (products) (knowledge, skills) –
resulting from completion of activities of
development interventions - Output
Development
bottlenecks
10
Types of Change
Impact
Outcomes
Outputs
Changes in the lives of people: realisation of
their rights
Institutional Change: values, laws – associated
with institutional performance, new institutions
Behavioural change: new attitudes, practices
Operational Change: products and services,
knowledge, skills resulting from completion of
activities of development interventions.
11
Definitions
Impact
and Primary
negativeeducation
long-termcompletion;
effects on
• Positive
Increased
e.g. Long lasting Changes in the situation (life) of
identifiable
groups produced&by
•Claim
Pupils
withpopulation
enhanced
lifea skills
Holders
(Children,communication
women, other vulnerable
development intervention, directly or indirectly,
groups) or
of their environment
• intended
BUT overloaded
secondary
education system
or unintended.
.
Outcome
The intended or achieved short-term and mediumInstitutional Change: values, ethic, rules, laws,
term effects of an intervention’s outputs, usually
practices,(particularly
performance
• Increased
retention
of girls)Outcomes
requiring
the collective
effort of partners.
Cognitiveenrolment
& Behaviour
change:
resulting from
•OrIncreased
(girls
and boys)
represent
changes in development
conditions which
knowledge,
skills
acquisition,
practices (individual,
• Improved
quality
of teaching/learning
occur
between
the completion
of outputs and the
society)
achievement of impact.
Output
Operational Change: products and services,
Two Primary
schoolsresulting
in each of
thecompletion
focus districts
Products
& services
from
of
Knowledge,
skills
resulting
from
completion
meet
Child-Friendlyinterventions.
School standard of
activities
of development
activities of development interventions
12
Definitions
Impact
Positive and negative long-term effects on
e.g.
Long
lastinglikely
Changes
the in
situation
(life) of
• identifiable
More
children
be in
living
a protective,
populationtogroups
produced
by a
Claim
Holders
(Children, women, other vulnerable
nurturing
environment;
development
intervention, directly or indirectly,
groups) or of their environment
intended or unintended. .
Outcome
The intended or achieved short-term and mediumInstitutional
Change:
values, ethic,
rules, laws,
• New
protection
policy adopted
by government
term effects of an intervention’s outputs, usually
practices,
performance
• Implementation
institutions
at national and
requiring the collective effort of partners. Outcomes
Or
Cognitive levels
& Behaviour
change:
resulting from
sub-national
created/
strengthened
represent changes in development conditions which
skills acquisition,
practices
(individual,
• knowledge,
National protective
environment
increasingly
in
occur between the completion of outputs and the
society)of CRC
line with recommendations
achievement of impact.
Output
Operational Change: products and services,
Products •Child
& services
resulting
from
completion of
protection
policy
drafted
knowledge,
skills
resulting from completion of
activities
of development
interventions.
•Key decision
makers convinced
and committed
activities of development interventions
13
Outputs: definition & key features
Outputs are deliverables
 Operational changes: new skills or abilities, the
availability of new products and services
 Must be achieved within the programme period
 Managers have a high degree of control
 If the result is mostly beyond the control or influence of the
programme or project, it cannot be an output
 Failure to deliver is failure of the programme or project
 4 to 6 outputs per CP outcome
 Unless under a joint programme, outputs are NOT
collective results
14
Test for “Outputs”:
1.Is it mostly (>80%) in your control?
2.Will you take singular accountability for it?
3.Will your funds be used to achieve this result
fully?
4.If achieved, will you take the credit for it?
5.If not achieved, will you take the blame for it?
If the answer is yes to all five questions above,
15
then it is an output. If not, think again.
CP Outcomes:
Definition & key features
The 2nd LEVEL RESULTS of UN system cooperation
 Represent changes in institutional performance or
behaviours
 A strategic contribution to the achievement of UNDAF
outcomes
 6 to 8 only per UNDAF outcome
 Accountability may be single agency, but is
increasingly shared
 Produced by the combined effects of the CP outputs
16
Assumption
Assumption
Probable/ less
risk
Inputs
Activities
Implementation partners
Less Probable/
more risk
Outputs
Plugging capacity gaps of
DB/ CH instts
Attribution
Assumption
Often quite
unpredictable/
even more risk
Outcomes
Improved performance of
DB and/ or behavioural
change in DB/CH
Impact
Change in realisation
of rights
Contribution
Decreasing Management Control..
Increasing external influences
Increasing difficulty in demonstrating attribution
17
An UNDAF programme
result at the outcome level.
Achievement depends on the
contribution of more than
one partner
A strategic UNDAF
result at the impact
level. Achievement
depends on joint
UNDAF outcomes plus
inputs from other
partners
Result
Joint or individual
agency result at the
outcome level
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
18
Joint or
individual
agency
results at the
output level
Result
Result
Result
Result
Result
Result
Achievement of UNDAF results depends on
the successful delivery of all partners’ contributions
Partnership
agreements needed?
Advocacy needed?
X
Result
X
X
Result
Result
X
Result
X
Result
Result
X
Result
X
Result
X
Result
X
Result
Result
Result
Result
Result
Result
Result
Result
Result
Result
Not delivered
19
Result
Result
Result
Result
Result
Result
A hypothetical UNDAF Results Framework
UNDAF outcome
25% reduction in new HIV infections by 2018
Existence of reliable
HIV surveillance
system
HIV surveillance
system designed
(WHO)
CP-level
outcomes
Enabling environment
exists that promotes
risk free behaviour
All large employers
have policy and
campaigns on HIV in
workplace
(ILO)
National and subnational bodies exist
that oversee
implementation of
AIDS prevention
activities.
(UNDP)
Voluntary and
confidential testing
and advice available
90% of people
practicing safe sexual
behaviour
Universal access to
condoms
Policy on VCT
agreed among major
stakeholders
(WHO)
High-risk population
groups know how to
prevent HIV infection
(Europ.Union)
Policy on condoms
agreed among major
stakeholders
(UNFPA)
Health workers able
to conduct VCT
(UNICEF)
90% of young
people 12-18 know
how to protect
themselves
(UNICEF)
Regular and
adequate supply of
condoms
guaranteed
(UNFPA)
Multi-media
campaign messages
reach 90% of all
people
(UNESCO)
CP-level
outputs
20
Key Principle # 1: Language
Action Vs Results Language
Action Language
services provider’s
Perspectives:
Unclear success
Criteria
Emphasis on
Processes
(activities)
Change Language
Rights holders/
Stakeholders’
perspective: They can
internalize and own it
success Criteria: clearer
Emphasis on change
(of capacities and status)
21
Refining Results
1.
Action Language
To strengthen the capacity of
teachers to teach life skills through
training on gender sensitive, child
friendly, learner directed learning
approaches
Change Language
1.
Results language to
emphasise future
condition and take out
information relating to
strategy or activities
2.
The capacity of teachers to teach
life skills is strengthened
3.
By 2010, the capacity of all life
skills teachers in the 10 districts
with the highest adolescent HIV
incidence rates is strengthened
2.
How many?
Everywhere? By the
year 4000? Be more
specific
All teachers in the 10 districts with
the highest adolescent HIV
incidence rates are able to teach
life skills to school children by 2010
3.
Bring the subject of
change to the front and
shift from passive to
active language
4.
22
To strengthen the capacity of UNICEF staff to
do RBP/M by undertaking a workshop…
The capacity of UNICEF staff is strengthened to
do RBPM by undertaking a workshop…
The capacity of select UNICEF staff in ICO is
strengthened to do RBPM by undertaking a
workshop
The capacity of select UNICEF staff in in ICO is
strengthened to do RBP/M
by undertaking a workshop
Select UNICEF
staff in
in ICO have the
capacity to do RBPM by December 2009.
Let’s use results language to
emphasise the future condition
we want to achieve.
All UNICEF staff
everywhere? Can you be
more specific? Are there
particular staff we should
emphasise?
We can take out information
that relates to either strategy
or activities.
Now, let’s try bringing the subject of
change to the front, and shifting from
passive to active language and add
time frame.
23
Quality Criteria for a Results statement
1. Is it stated using change language?
2. Is the scale/scope realistically within the control of you
and your partners?
3. Is there an If-then causality between output and
outcome and are sum of outputs sufficient to achieve
higher level outcome?
4. Make sure output does not repeat the outcome
statement and is not a completed activity (training
conducted, workshop completed; survey implemented;
law reviewed)
5. Make sure it is SMART – Specific, Measurable,
Achievable, Relevant, Time bound
6. Take reference to strategy out of sentence
7. To what extent will the most vulnerable/disadvantaged
participate in and benefit from the output;
8. Has the gender dimension been considered?
24
Output in the context of human rights
How?
Inputs
Activities
Implementation partners
What do we want to change?
Outputs
Plugging capacity gaps of
DB/ CH instts
Outcomes
Improved performance of
DB and/ or behavioural
change in DB/CH
Why?
Impact
Change in realisation
of rights
26
CAPACITY ANALYSIS
Components
1. Authority
2. Responsibility/ motivation/ will to do
3. Resources
4. Triple A decision-making capability
5. Communication capability
27
The challenge of capacity development:
Working towards good practice (OECD paper - 2006)
Individual level
(experience, knowledge & technical skills)
Systemic
factors, i.e.,
relationships
Organizational level
between the
(systems, procedures & rules)
enabling
environment,
organization
s
and
Enabling environment
individuals (institutional framework, power structure & influence)
Successful capacity development requires not only skills &
organizational procedures, but also incentives & good
governance
Influen
ces by
means
of
incenti
ves it
creates
28
UNDG Capacity Assessment Framework – a ThreeDimensional Cube (2008)
Technical
Capacities
Functional Capacities
Policy Design Resources Programme Monitoring,
Situation
& Strategy
& Budget
& Project
Evaluation &
Analysis
Formulation Allocation Implementation Learning
Core Issues
Public Sector
Accountability
Access to Information,
Development Knowledge and
Technology
Inclusion, Participation, Equity
and Empowerment
External/International
Relations
ts
oin
try
n
E
of
P
Human Rights Based Approach and Gender Equality Are
Embedded in Every Capacity Assessment
g
blin ent
a
n
E
m
i ro n
v
n
E
’l
Org
al
idu
v
i
nd
I
29
The UN System and Capacity Development
Background Paper (2006)
1. Don’t rush
- long-term process.
2. Respect the value systems and foster self-esteem self-esteem is at the root of capacity and empowerment.
3. Scan locally and globally; reinvent locally
blueprints.
- there are no
4. Challenge mindsets and power differentials - capacity
development is not power neutral.
5. Think and act in terms of sustainable capacity
outcomes.
30
The UN System and Capacity Development
Background Paper (2006)…
6. Establish positive incentives.
7. Integrate external inputs into national priorities,
processes and systems - external inputs need to
correspond to real demand
8. Build on existing capacities rather than creating new
ones - this implies the use of national expertise as prime
option.
9. Stay engaged under difficult circumstances - the weaker
the capacity the greater the need.
10.Remain accountable to ultimate claim holders - any
responsible government is answerable to its people, and
should foster transparency as the foremost instrument of
public accountability.
31
Examples of Indicators
Goal
Impact indicators
Purpose
Outcome indicators
Infant mortality rate,
under 5 mortality rate
Objectives
results
Number & % of child diarrhea
cases treated with ORT
Results monitoring
Number & % of mothers knowing
about & having access to ORT services
Outputs
Output indicators
Activities
Process indicators
Inputs
Input indicators
Strategies
means
Number of health personnel trained in
ORT, number of media ads produced, etc
Funding amounts ($), trainers
Implementation (person-months), ORT supplies (numbers),
monitoring
etc
Project Logical Framework
Business Counseling Example
Improved business
performance
These owners gain knowledge and
change practices resulting in
A variety of educational activities are
provided to business owners who
participate
Agency invests time and resources
University of Wisconsin-Extension, Program Development and Evaluation
33
Example: Water quality
University of Wisconsin-Extension, Program Development and Evaluation
34
PROGRAM LEVEL
Macro
RESULTS CHAIN
Long term
Millennium
Development Goals
Country Program Results
Financial, Knowledge,
Sector Results
Budget Resources
& Projects
Micro
ATTRIBUTION
Short term
Low
Impacts
Outcomes
Outputs
Inputs
High
Results Framework
Programme
Vision
Strategic
Strategic
Strategic
Outcome
Outcome
Outcome
Intermediate
Intermediate
Outcome
Outcome
Intermediate
Outcome
Intermediate
Outcome
Outputs
Partner 3 ($)
Outputs
Partner1 ($)
Outputs
Partner 2 ($)
Outputs
Partner 4 ($)
Outputs
Partner 5 ($)
36
Definitions
Impact
Outcome
Output
Positive and negative, primary and secondary
long-term effects produced by a
development intervention, directly or indirectly,
intended or unintended.
The likely or achieved short-term and
medium-term effects of an intervention’s
outputs.
The products and services which result from
the completion of activities within a development
intervention.
Result Chain: Definitions
the overall and long-term
effect of an intervention; it is
the ultimate improvement/
changes in people’s lives;
intended/ unintended e.g.
Improved access to
justicernance
Tangible, time-bound
products resulting from
completion of activities; are
under the control of the
project e.g. Policy makers,
CSOs and judiciary officials
trained and able to apply human
rights treaties and instruments
inputs are means
mobilized to carry out
activities
e.g. funds, staff time and
other resources
IMPACT
Institutional and behavioral
changes in the real world
triggered by the outputs;
outcomes are not under the
OUTCOME control of the project any more
(in contrast to outputs)
e.g. Adherence to international
human rights instruments and
treaties.
activities are the various
OUTPUTS steps taken to carry out a
project
ACTIVITIES
INPUTS
e.g. Material on raising
awareness on human rights
developed and translated
into local languages
Organize training workshops
Outcome Formulation: An example…
Basic Services
Action Statements:
1. Increase participation of vulnerable groups &
their access to basic health
2. Enhance safe and equal access to quality
education, health
3. Increase the safe and equal access to primary
and secondary education
Environment
Action Statements:
1. Strengthen capacity to comply with EU/UN
standards and environmental conventions
2. Support government/CSOs to develop and
implement sound agricultural and environ. policies
3. Ensure sustainable use of natural resources
Good Governance
Action Statements:
1. Support people-oriented governance through
decentralization and local capacity building
2. Build the capacity of government structures to
be transparent and accountable
Types of Change
Impact
Changes in the lives of people
Outcome
Institutional Change: values, ethic, rules, laws –
associated with to institutional performance, access..
Behavioural change: knowledge, skills, practices
(individual level)
Output
Operational Change: provision of goods and
services
To achieve results you need
to..
 Set strategic intents which
provide a focus for action
 Formulate “SMART” results
 Identify baselines
 Set intermediary targets
 Select performance
indicators
 Be determined to do it.
Selection Criteria for Results
S
Specific
M
Measurable
A
Achievable
R
Relevant
T
Time bound & Trackable
A Typology for RBM: Poverty Reduction (2)
Results
Like…
Impact
Poverty reduced
Outcome
Employment and
income generation
increased
Outcome
3000 new small
enterprises developed
in poorest provinces
Output
7 model business
incubators operational
in poorest provinces
Activity
- Acquire facilities
- Staff training
- Micro-credit provision..
Focus
Human!
Institutional/
Behavioural
Institutional/
Behavioural
Operational/
skills, abilities,
products &
services
@
Timeframe
5-10 yrs
5 yrs
5 yrs
<5 yrs
<1 yr
Annual Workplan
EXPECTED OUTPUTS
And baseline, indicators
including annual targets
Output 1
Baseline:
Indicators:
Targets:
Related CP outcome:
PLANNED
ACTIVITIES
List activity results
and associated
actions
1. Activity Result
-Activity action
-Activity action
2. Activity Result
-Activity action
-Activity action
3. Activity Result
- Activity action
- Activity action
4. Activity Result
-Activity action
-Activity action
Output 2
Baseline:
Indicators:
Targets:
Related CP outcome:
Output 3
Baseline:
Indicators:
Targets:
Related CP outcome:
TOTAL
1. Activity Result
-Activity action
-Activity action
2. Activity Result
-Activity action
-Activity action
TIMEFRAME
Q1
Q2
Q3
PLANNED BUDGET
Q4
RESPONSIBL
E PARTY
Funding
Source
Budget
Description
Amount
Harmonised AWP Monitoring Tool
• Regional (or Country) Programme Component:
_________________
• Implementing Partners: _________________Year: ____
Expenditures Results
RP
Planned
Progress
(from FACE
Outputs,
Activity
(UNDP:
towards
form or DEX Delivera- achieving
indicators,
expenditure)
Annual
Targets
Output 1
Output 2
(UNDP:
Deliverables)
bles) of
Activities
RP
Outputs
Updating
Risks,
Issues &
Lessons
Result Chain
What are Good Indicators?
SPECIFIC
• In terms of quantity, quality, time, location, target groups, baseline and
target for the indicator
MEASURABLE
• Will the indicator show desirable change?
• Is it a reliable and clear measure of results?
• Is it sensitive to changes in policies & programmes?
• Do stakeholders agree on exactly what to measure?
ACHIEVABLE
• Are the result (s) realistic and based on risk assessment, partnership
strategy and other factors contributing to the underlying result
RELEVANT
• Is it relevant to the intended result?
•
Does it reflect the expectations and success criteria for change in the
target groups?
TRACKABLE/TIMEBOUND
• Are data actually available at reasonable cost & effort?
• Can proxy indicators be used?
• Are data sources known?
What are Indicators?
A quantitative or qualitative variable that allows the
verification of changes produced by a development
intervention relative to what was planned
(UNDG Harmonized Terminology, 2003)
In other words ….
Means of measuring what actually happens against what
has been planned in terms of quantity, quality and timeliness,
for every level of result.
Purpose
of
Indicators
Indicators ‘indicate’
that change is
happening or not
happening.
They can….
Clarify the scale and scope of a result in the
results framework
Demonstrate progress when things go right
Provide early warning when things go wrong
Assist in identifying changes that need to be
made in strategy and practice
Inform decision making
Facilitate effective evaluation
Baseline
A baseline is the situation just before, or at
the outset of a new program, project,
service or operation against which
progress can be measure or comparisons
can be made as part of monitoring and
evaluation.
Effective monitoring is nearly impossible
without an established baseline.
Targets
A target is an explicit statement or result
derived for an indicator over any specified
time period (to be provided at the level of
output, outcomes and impact).