Transcript Do we know how to create jobs? Evaluation lessons from a Systematic Review

Do we know how to create jobs?
Evaluation lessons from a Systematic Review
by Michael Grimm and Anna Luisa Paffhausen,
University of Passau, Germany
Commissioned by
KfW Entwicklungsbank Evaluation Unit
Disclaimer: The views and opinions expressed in the Systematic
Review are those of the authors and do not necessarily reflect
the official policy or position of any agency of the German
Development Cooperation.
Systematic reviews
...a way to overcome the problem of
limited external validity of experimental
(RCT) and quasi-experimental
outcome/impact measurements.
Disclaimer: The views and opinions expressed in the Systematic
Review are those of the authors and do not necessarily reflect
the official policy or position of any agency of the German
Development Cooperation.
KfW presents
itself / February
November2014,
2013 Paris
OECD DAC
EvalNet 12-13
2
Search strategy and intervention categories
2,275 record identified
through database searching
Identification
Screening
1,924 records screened
after duplicates removed
1,785 records excluded
(mostly because not related to our research question)
Eligibility
Intervention types
Included
139 studies assessed
for eligibility
55 studies included
Incl. 27 RCT studies
93 impact estimates
Access to finance and insurance
•
26 studies / 13 RCTs
84 studies excluded
5 not obtained
15 studies ongoing
20 additional records identified
through other sources
Entrepreneurship training
•
20 studies / 16 RCTs
Business development services
and targeted subsidies
Improvements of business environment: Incentives to formalise
•
•5 studies / 1 RCT
10 studies / 1 RCT
KfW presents
itself / February
November2014,
2013 Paris
OECD DAC
EvalNet 12-13
Not adding
up to 55
because
some studies
cover more
than one
intervention
type
3
Key findings on employment
26 studies, 40 Treatment effects: 13 positive, 2 negative, 25 insignificant
Access to finance
and insurance
•
•
•
•
No large employment effects, particularly not in micro-enterprise.
More business creation than hiring of new workers
Interventions mostly do not have job creation as major objective (but rather
income stabilisiation, higher family income).
Employment effects larger if owner is male.
20 studies, 31 treatment effects: 9 positive, 1 negative, 21 insignificant
Entrepreneurship training
•
•
•
Successful training does not necessarily lead to employment generation: it can
even lead to closing down unsuccessful enterprise.
Training should be substantial (about 1 year, once a week) and specific.
Tackling finance impediments at the same time seems to make employment
effects more likely.
10 studies, 7 positive or mixed, 1 negative (minimum wage), 1 insignificant
Business development services and
targeted subsidies
•
•
•
Targeted subsidies for hiring a worker or for innovations/R&D can contribute to
employment generation
Subsidies and services should be demand (not supply) driven, tailor-made and
focussed.
Expensive interventions
5 studies, employment effect after formalisation 4 positive, 1 insignificant
Business environment: Incentives
to formalise
•
•
•
Only few (larger) and new firms formalise; for microentrepreneurs often formality
has more disadvantages than advantages.
Formalisation, where it worked, had only modest employment effects.
Better services that firms gain access to with formalisation offer incentives to give
up informality and might contribute to job creation as well.
KfW presents
itself / February
November2014,
2013 Paris
OECD DAC
EvalNet 12-13
The method bias
Studies that are based on RCTs show a lower share of
significantly positive employment effects than studies that rely
on quasi-experimental methods.
 Do quasi-experimental studies
over-estimate employment
effects due to un-eliminated
biases?
 ...or are employment effects
small in RCT-measurements
because this method is very
often applied to rather small
programmes in relatively poor
areas?
KfW presents
itself / February
November2014,
2013 Paris
OECD DAC
EvalNet 12-13
5
Implications for development policy, evaluation and research
Policy
Evaluation and research
› It is a long way from policy inputs to
› Evidence is still sketchy,
employment impacts: it seems easier to
achieve effects on management practices,
sales or profits than employment effects.
› A major push is needed to have
employment impact.
› Many of the interventions included in the
review first of all strive for income
stabilisation and poverty reduction, and not
for employment creation.
› Targeting seems to be key.
› It seems to be easier to create new
businesses than to foster the growth of
existing firms.
KfW presents
itself / February
November2014,
2013 Paris
OECD DAC
EvalNet 12-13
particularly for Sub-Saharan Africa and Asia.
› There is almost a complete lack of evidence
on long-term effects and cost-effectiveness.
› A dilemma: For the inclusion into a systematic
review, studies have to meet highest
methodological quality standards. However,
not all programmes are suited for (quasi)experimental evaluation designs
How to deal with the dilemma:
 Do not neglect the findings of other types
of evaluations
 ...until research comes up with new quality
standards that cover a wider range of
programme designs
6
Thank you!
KfW presents
itself / February
November2014,
2013 Paris
OECD DAC
EvalNet 12-13
7