Transcript MSI Seminar Presentation: Monitoring & Evaluation Methodology

MONITORING AND EVALUATION
METHODOLOGY
KEVIN P O’KELLY
Draft Report

Introduction
 Definitions

Social Inclusion as a European Issue
 OMC
 NAPs
 Scope of MSI Project

Why Mainstreaming?
Draft Report

Poverty, Social Inclusion and Public
Policy

Participative Methodology

Monitoring and Evaluation
Monitoring and Evaluation
Structure of the chapter:
 Define ‘Monitoring;’ and ‘Evaluation’
 Theory of Evaluation
 Designing an evaluation scheme (research
design)
 Indicators
 Evaluating MSI
MONITORING
Monitoring or Process Evaluation:
 Carried out during implementation
 How, Why and under what conditions?
 What happens during implementation?
 Is implementation in line with original
design?
Evaluation
Types of Evaluation
 Impact or summative
 Outcome
 Variation
 Counterfactual
What is Evaluation?
A systematic assessment of the
operation and/or the outcomes of a
programme or policy, compared to a
set of explicit or implicit standards,
as a means of contributing to the
improvement of the programme or
policy (C H Weiss, 1998)
What is Evaluation?
Evaluation
Monitoring
Appraisal
Feedback
Rationale
Objectives
UK Treasury Green Book
What is Evaluation?
F
Policy
Review
FEEDBACK
E
Policy
M
E
O
Rational
A
O
M
A
ACTORS




Political / Policy level
Administration / Management
Service providers
Target Groups / Recipients
Design of Evaluation
Evaluation Questions



Does the policy work?
Why does it work?
Why (how) should policies work?
(Robert Walker – June 2004)
Infrastructure for evaluation
7 What worked?
6
Is there a problem?
2
Has the policy
worked?
1
What policy
would work?
POLICY
5
3
Will this
policy
work?
Is this policy
working?
4
Can we make this
work?
Evaluation Questions




Factual
Behavioural
Attitudinal
Knowledge
Design of Evaluation

Twelve Steps
Advisory Committee
 Resources





Selection of evaluators
Key questions
Methodology
How to collect data
Questionnaire / interview guidelines
Design of Evaluation






Target sample
Field work
Analysis
Meta-analysis
Write up findings
Publication / dissemination
Policy Indicators

Social Policy Committee (Laeken)
indicators





Low incomes households
Long-term unemployed
Low education levels
Health status
‘In-work’ poor
Policy Indicators

EAPN Indicators






Employment
Income levels
Housing
Health
Education
Participation and identity
 Definition
Policy Indicators
Participation and Identity - A Definition:
The percentage share of the population
with an income below 60% of the median
(national poverty level) that are members
of or connected with:
(a range of social, community and
cultural activities)
Evaluating MSI




Mainstreaming is a process
Political commitment
Involve ALL key stakeholders
Realistic expectations
Evaluating MSI






Design issues:
What target groups?
What outcomes?
Quality of data
Comparison of small and large units
Collection of data at point of delivery
(local level)
Challenge of ‘Silo’ policies
Evaluating MSI







Access to data
What level of governance is
mainstreaming implemented?
Political environment / decision-making
Structures
Link between governance roles
Culture
Different criteria for success
MSI Question
Does Mainstreaming of Social Inclusion
have an impact on the policy process
and outcomes? If so, can it be
measured?


European level: NAPs/incl. & OMC
Implementation level of NAPs/incl.
Stephen Donnelly Paper

Mainstreaming issues
 Any measurement of mainstreaming will
effectively be a measurement of qualitative
processes … the ultimate purpose of
mainstreaming is to produce measurable
poverty reduction outcomes.
 A key challenge in attempting to determine how
far poverty reduction activities are
mainstreamed centres on the subjectivity of any
measurement tools that are put in place
Stephen Donnelly Paper
Measuring ‘Mainstreaming’ is subjective!
How to define:
 Political will /leadership
 Partnership
 Ownership
 Cross-departmental working?
Stephen Donnelly Paper



‘Positive action’ initiatives are NOT
mainstreaming
However, mainstreaming doesn’t preclude
‘positive action’
‘Poverty Proofing’
Stephen Donnelly Paper
A number of elements which are
fundamental to mainstreaming:
 Leadership
 Structures
 Capacity and skills
 Community participation and
engagement
 Research and evaluation
Why
Mainstreaming?
Stephen Donnelly Paper






Draft Measurement Framework:
Political Leadership and sponsorship
Executive leadership and strategies
Capacity
Structures
Data, research and evaluation
Community engagement and participation
Stephen Donnelly Paper
Poses the question: is
social auditing / theory of change
an alternative to Mainstreaming?
Measuring Mainstreaming



Qualitative?
Quantitative?
Combination?
EVALUATION TYPOLOGY II FOR MAINSTREAMING SOCIAL INCLUSION
CONTEXT
(EU; National; Regional; Local // Economic; Demographic; Social; Cultural; etc.)
Mainstreaming
Social Inclusion
Definition
REQUIRMENTS FOR
EVALUATION
 Clear Policy Objectives
 Clear ‘Theory of Change’
 Clear Evaluation Objectives
Features
Process
 Cross-cutting
 Inputs (resources)
 Policy development
 Participation
 Monitoring and evaluation
 Political commitment
 Organisation of resources
 Outputs
EVIDENCE
Case studies by
 Theme
 Country
 Governance level
 Indicators
EVALUATION QUESTIONS
(Robert Walker)
KEY QUESTIONS EVALUATION OF MSI:
 What would be good evaluation questions on the
process of mainstreaming (Mst.)?
 Can we build a scenario for an evaluation framework
• a) identify & b) measure the impact of Mst.?
to
 Do we have the tools to analysis the process and
measure the impact? If ‘no’, how do we get the tools?
Revised
Research
Question
Outcomes
 Can we identify evaluation processes of Mst. in the
different Member States?
 Why Mst? Is it better?
 Meta-analysis
(JIMs, JIRs)
 Theory of change
Scale of features
Interviews with key
actors
SCENARIOS OF EVALUATION FRAMEWORK
 What worked?
 How did it work?
 Has the policy worked?
 How did it (not) work?
 Is this policy working?
 How is it working?
 Is there a problem?
 What is the problem?
 What policy would work?
 How would it work?
 Will this policy work?
 How will it (not) work?
e.g. Is Mts. a process / tool or a policy?
No counterfactual
 Can we make this policy work?
 How can we make it work?