Transcript National strategies for Roma Inclusion – the role of data

MONITORING AND EVALUATION
OF NATIONAL STRATEGIES FOR ROMA INCLUSION:
WHAT DATA AND FOR WHAT PURPOSE?
Andrey Ivanov, Senior Policy Advisor,
Human Development and Roma Inclusion cluster, UNDP BRC
MONITORING AND EVALUATION 101
Basic typology
 Monitoring (the process)
 Evaluation (of the results)
 Intermediary

or final
Using indicators
 Input
 Output  Outcome  Impact
Applied at different levels
 Of
the National strategy
 Of the Action Plans
 Of Individual interventions
Monitoring what
determines the kind
of data and the kind
of indicators used
THE STRATEGIES: HOW TO GET RESULTS

Having a National Strategy drafted is the beginning, not the end. It
needs to be matched by




In the case of the strategy, for M&E we need





National Action Plans (usually covering 2 years periods and regularly
updated)
Local action plans
Sector specific and integrated projects
Clear targets – numerical expression of the objectives
Adequate indicators – the definition of the target (how do we measure
whether the objective was reached)
Quantitative baseline – the starting point against which the
progress/regress is be quantified (the value of an indicator at to)
Milestones – intermediary targets on the way to the general
target to keep track of progress (the value of the indicator
at t2, t4, t6)
The lower you go, the higher the chances for real inclusion of Roma in
the process
DIFFERENT RESULTS AT DIFFERENT LEVELS

National strategy
Long-term change in the situation of the target group
 Difficult to attribute results (but not impossible)


National action plan
Closer link between inputs and outcomes
 Clear objectives (that are the strategy’s milestones


Local action plans
Direct link to project outputs
 Clear territorial dimensions


Individual interventions

Counterfactual possible although difficult
OPEN QUESTIONS

What targets for individual priority areas?
 Roma

specific or general?
What baseline?
 2004?

2011? 2013?
What source of data?
 Data
availability determines the indicators or the
other way around?
What milestones?
 The link to individual OPs

EXAMPLE: ENROLLMENT RATE
ANOTHER OPEN QUESTION: WHO’S ROMA
Politically sensitive (incl. misuse of data for
political purposes)
 Legal (data protection) or ethical
considerations (privacy and fear of stigma)
constrains
 Insufficient attention to comparability across
countries, sub-regions, ethnic groups
 The crucial question: what to put in the
denominator of an indicator?
 The nightmare answer: whatever serves the
purpose…

WHO IS ROMA? POSSIBLE OPTIONS


Self-identification
Outside (‘imposed’) identification




By non-Roma
By Roma
Combined (multi-stages) – used in the surveys of
UNDP (2004 and 2011) and of FRA (2011)
Crucial decision to be made: are we addressing “all
Roma” – or “Roma at risk of marginalization”? The
answers is both politically and policy loaded.
POSSIBLE SOURCE OF DATA FOR M&E
The data set of Roma vulnerable to marginalization generated from the
UNDP/WB regional survey that is part of EU Roma Pilot Project funded
by DG REGIO and from FRA Roma Pilot Survey:
 Monitoring fundamental changes possible (but not short-term
fluctuations). Suitable for National Strategy evaluation
 Most indicators have a base-line populated by data from the survey
conducted in 2004 by UNDP
 The “best game in town” (because it’s the only one…)
Caveats:
 Still a survey (a sample is always a sample)
 Expensive, provides data on “Roma vulnerable to marginalization” –
and not on “Roma in general”
Other options
 Roma boosters in HBS
 Longitudinal surveys
SURVEY DATA IS… SURVEY DATA
GOING BEYOND ETHNIC IDENTITY

Be pragmatic - don’t be obsessed by (don’t ask)
unanswerable questions like “Who’s Roma?”


Give priority to socio-economic status


But don’t dilute the task of Roma inclusion either
But still keep ethnic identity and specifics in sight
Stick to territorial characteristics




Most of the vulnerable Roma live territorially in separate
(segregated) communities
Territorial mapping of those communities is possible
Once a detailed map of Roma-dominated communities is
available, it will be possible to correlate ethnic characteristics
with territorial tags (individual’s address)
This will allow monitoring a standard set of indicators for a
population living in an area with ***% of Roma
THE BENEFITS OF TERRITORIAL APPROACH





Makes possible to identify the absolute number of the
population and not only a percentage
It can be an option solving the problem of individual
respondents refusal to declare ethnicity in the census
or to declare different one
Less susceptible to political fluctuations
Is more comprehensive in terms of social inclusion
(targeting vulnerability per se)
It grasps the marginalized, visibly excluded segment of
the Roma population


Actually reflects the fundamental logic of inclusion
(including the excluded, not those included already)
Is best for ensuring that control groups (non-Roma
living in the same area) are also included
AN OPTIMAL COMPROMISE
One approach cannot serve all purposes
 Apply different data sources for different
planning frameworks

 National
Strategy – EU-wide survey (representative
of… - a matter of political compromise)
 National Action Plans – territorially-focused
mapping
 Individual interventions – project outcome
evaluation

Integration of the three levels requires clear
milestones in strategies and action plans
CONCLUSIONS






Integrate the monitoring functions into the entire
implementation chain of the strategy
Don’t rely on one source of data and give priority to
territorial approaches
Include clear milestones in National Strategies that
would serve as a link to the National action plans and
OPs
Compete the entire vertical planning and M&E
architecture (strategy  plan  call for proposals 
interventions)
Go beyond poetry in Operational Programs evaluation
building the latter bottom up
Be aware: keeping evaluations vague means keeping
them fake