Transcript Musings on Governance Indicators, and some pathways forward… Daniel Kaufmann The World Bank Institute Presentation at Seminar on The Empirics of Governance, May 1-2, 2008,

Musings on Governance Indicators,
and some pathways forward…
Daniel Kaufmann
The World Bank Institute
Presentation at Seminar on The Empirics
of Governance, May 1-2, 2008, World
Bank, Washington DC
1
Overview
• Preamble: Many extremely valuable efforts…
• Measuring governance/institutional quality
– some conceptual / definitional musings
– what do we measure? rules vs. outcomes
– whose views? experts vs. survey respondents
– measurement error
– disaggregation, ‘actionability’, & aggregation
• Constructively Moving forward…
2
Do we know what governance is?
• Ok: No Agreed standard or Unified Theory (AUT) of governance
• Yet: Absence of AUT is also the case with most any field in
social science/economics fields… [& physics..?]
• And: Absence of AUT in many fields hasn’t deter measurement
• In governance (as in other fields), in spite of no AUT: there is a
scholarly body of theorizing on governance & institutions
• Further: various definitions have been put forth over the years,
mostly they are variants of each other – but we opt against
either anorexic or tautologically ‘fat’ definitions – ‘broad’ is ok
• Distinguish between pre-requisites for measuring governance,
vs. hypotheses on whether, what, and extent of governance
mattering for development outcomes (though iterative process)
• Key question: do conceptual or definitional differences make a
1st or 2nd order difference to the measured indicators (or are
other factors more important)?
3
Definitional Issues, in more detail (1)
• Makes sense to avoid overly fat definitions
– risks making links from governance to development
tautological (think about North quote)
– risks lack of focus in what we measure
• e.g. Ibrahim Index of African Governance averages
together things like corruption, inflation, and per capita
GDP
• Also makes sense to avoid overly narrow definitions
– constitutions  institutions!
– de facto rules matter more than their de jure counterparts
• Don’t overstate degree of definitional disagreement, most
definitions encompass capable states, operating under rule
of law, and accountable to citizens
4
What do we mean by “Governance”, Cont’d (2)?
• World Bank (1992): "Governance is the manner in which
power is exercised in the management of a country's
economic and social resources for development“
• World Bank (2007) definition: "...the manner in which public
officials and institutions acquire and exercise the authority to
shape public policy and provide public goods and services"
• WGI Definition (1999): "...the traditions and institutions by
which authority in a country is exercised. This includes the
process by which governments are selected, monitored and
replaced; the capacity of the government to effectively
formulate and implement sound policies; and the respect of
citizens and the state for the institutions that govern economic
and social interactions among them."
5
A Fundamental Challenge
in Measuring Governance
• The ‘true’ level of governance is inherently
unobservable (or, at least, in some dimensions,
extremely hard to measure)
• Unavoidably: any measure of governance and
investment climate are proxies of the ‘true’ level
• Cats, mousetraps and bathwater filters: the above
means that we need to take very seriously: i)
triangulation; ii) margins of error; iii) moving from
objective-subjective distinction to more meaningful
ones…
• Further, different objectives means different tools
are more appropriate
6
A Taxonomy of Governance Indicators: moving away from the
subjective-objective divide to rules vs outcomes-based
Whose Opinion?
About What?
Rules
Broad Specific
Experts
Lawyers
Commercial Risk Rating Agencies
Non-Governmental Organizations
Governments and Multilaterals
Academics
Outcomes
Broad
Specific
DRI, EIU, PRS
HER, RSF, CIR, FRH
CPIA
DPI, PIV
GII, OBI
PEFA
DB
GII
DPI, PIV
Survey Respondents
Firms
Individuals
Aggregate Indicators Combining Respondents
ICA, GCS, WCY
AFR, LBO, GWP
TI, WGI, MOI
7
Legend for Taxonomy
Countries
Covered Frequency
Code
Name
AFR
CIR
CPIA
DB
DPI
DRI
EIU
FRH
GCS
GII
GWP
HER
ICA
LBO
MOI
OBI
PEFA
PIV
PRS
RSF
WCY
Afrobarometer
18
Cingranelli-Richards Human Rights Dataset192
Country Policy and Institutional Assessment136
Doing Business
175
Database of Political Institutions
178
Global Insight DRI
117
Economist Intelligence Unit
120
Freedom House
192
Global Competitiveness Survey
117
Global Integrity Index
41
Gallup World Poll
131
Heritage Foundation
161
Investment Climate Surveys
94
Latinobarometro
17
Ibrahim Index of African Governance
48
Open Budget Index
59
Public Expenditure and Fiscal Accountability42
Polity IV
161
Political Risk Services
140
Reporters Without Borders
165
World Competitiveness Yearbook
47
Every 3 years
Annual
Annual
Annual
Annual
Quarterly
Quarterly
Annual
Annual
Every 3 years
Annual
Annual
Irregular
Annual
Every 3 years
Annual
Irregular
Annual
Monthly
Annual
Annual
Link
www.humanrightsdata.com
www.worldbank.org
www.doingbusiness.org
http://econ.worldbank.org
www.globalinsight.com
www.eiu.com
www.freedomhouse.org
www.weforum.org
www.globalintegrity.org
www.gallupworldpoll.com
www.heritage.org
www.investmentclimate.org
www.latinobarometro.org
www.moibrahimfoundation.org
www.openbudgetindex.org
www.pefa.org
www.cidcm.umd.edu/polity/
www.prsgroup.com
www.rsf.org
8
www.imd.ch
What To Measure: Rules or Outcomes?
• Rules-based de jure measures capture formal rules,
institutions, and processes
– presidential vs. parliamentary system
– first-past-the-post vs. proportional representation
– does a statutorily-independent election monitoring
agency or anticorruption agency exist?
• Outcome-based de facto measures capture how these rules
are implemented and their effects in practice
– is election monitor politicized?
– do firms perceive corruption is a problem?
– do teachers show up at school?
No “bright line” separating the two, think of as a continuum 9
Rules-Based Indicators
• Main virtue is clarity: does a rule exist? has legislation been
passed?
– obvious appeal to gov’ts / donors who want to monitor
progress on governance
– reflected in language of “actionable” indicators
• Three main drawbacks
– easy to overstate clarity: measurement of all but most
basic rules requires careful judgment – virtually no truly
“objective” indicators exist
– links from (highly detailed) rules to outcomes not always
clear
• which “actionable” indicators are “actionworthy”?
• risk of “reform illusion” or “teaching to the test”
– obvious potential for gaps between de jure rules and de
facto outcomes
10
Outcome-Based Indicators
• Majority of governance indicators focus on measuring de facto
outcomes (governance outcomes, that is…)
– budget outcomes relative to budget plans? (PEFA, OBI)
– fraction of contract value paid as bribe? (GCS)
– is corruption in government widespread? (GWP)
• Main virtue is that these measure relevant governance
outcomes as seen by stakeholders, including informality
• Drawbacks:
– (broad) outcome measures hard to link back to specific
policy interventions (mirror image of previous problem)
– (poorly-designed) questions can be vague and open to
interpretation
• avoid generic questions, focus on experiential ones
• provide benchmarks for respondents (e.g. vignettes)
11
Subjective and Objective Measures of
Ease of Business Entry: OECD/NIC Sample
Difficulty of Starting a Business (EOS)
High
5
4
3
r = 0.51
Low 2
0
40
80
Number of Days to start a Business (DB)
12
120
Subjective and Objective Measures of Ease of
Business Entry: Developing Country Sample
Difficulty of Starting a Business (EOS)
High
Low
7
r = 0.24
5
3
1
0
40
80
120
160
13
Number of Days to start a Business (DB)
Are Elections Free and Fair? Global Integrity
vs. Global Corruption Barometer Survey
Global Integrity Index
De Jure and De Facto
Indicators of Elections Integrity
120
y = -3.14x + 85.69
De Jure
2
R = 0.00
100
USA
80
ISR
GHA ZAF
ROM
YUG
BGR
GTM
ARG
NIC
NGA
60
40
ETH
PHL
IND
PAK
MEX
RUS
De Facto
KENIDN
SEN
y = 23.09x + 55.30
R2 = 0.19
20
0
0
0.2
0.4
0.6
0.8
1
Voice of the People Household Survey
(Are Elections Free and Fair?)
Source: Center of Public Integrity, 2006 - http://www.globalintegrity.org/, and Gallup International.
17
Whose Views To Rely On
-- Experts or Survey Respondents?
• Expert assessments produced by wide variety of
organizations, e.g.
– Multilateral organizations (World Bank CPIA)
– Commercial risk rating agencies (EIU, DRI, PRS,
WMO)
– NGOs (Freedom House, Global Integrity)
• Several large cross-country surveys
– of firms (GCS, WEF, BEEPS)
– of households (Afrobarometer, Latinobarometer,
Gallup World Poll)
18
Asking the Experts:
• experts natural respondents for certain questions
– details of budget or regulatory procedures not
“common knowledge”
• are experts “biased”?
ideological biases: do “right-wing experts” like “right
wing governments”?: difference between expert
assessments and surveys not highly correlated with
political orientation of country’s government
– business-elite biases: do businesspeople just want
low taxes and no regulation?: business and nonbusiness views on governance often correlate well
across countries
• do experts engage in “group-think”? -- in GM V: experts no
more correlated among themselves than w/ surveys untainted
by group-think, so little evidence of correlated errors
19
Asking Households and Firms
• Main advantage is we get information from the ultimate
beneficiaries (victims) of good (bad) governance
– their perceptions matter because they act on them
• Main disadvantages
– costly to do regular cross-national surveys
• concerns about representativeness (but far less than for
single-expert assessments)
– badly-designed questions are open to interpretation
• but replace with good questions
• generic problem for all types of indicators
– respondent reticence
• use random response methods to alleviate
-- type of political regime may affect responses (enterprises)
20
-- Thus, triangulation is paramount – looking
at various rules and outcome-based
indicators, drawn from experts and surveys
-- Further, studying what matters, what is
action-worthy is important
22
Anti-Corruption Agencies (GII rating) associated w/ extent of Corruption (as reported by individuals)?
Global Corruption Barometer 2006:
Fraction of Respondents Answering No to
"Is Corruption Widespread?"
0.8
r = 0.22
ISRAEL
SOUTH
AFRICA
UNITED STATES
0.7
ARGENTINA
ROMANIA
SERBIA
KENYA
INDIA
PHILIPPINES
BULGARIA
RUSSIA
SENEGAL
INDONESIA
0.6
MEXICO
PAKISTAN
0.5
40
50
60
70
80
90
100
Global Integrity Index 2006: "De Jure" & "De Facto" Component of
Anticorruption Institutions
Source: Center of Public Integrity - http://www.globalintegrity.org/, and transparency International.
23
Anti-Corruption Agencies (GII) associated with extent of Corruption (as reported by
enterprises)?
Global Competitiveness Survey 2006:
Composite Corruption Index
0.8
ISR A EL
STAA T ES
SOUUTNHITAED
F R IC
IN D IA
EGY PT
0.6
B U LGA R IA
M EXGU
ICA
OT EM A LA
B R A Z IL
M ONTENEGRO
A R M EN IA
0.4
SER B IA
ET HIOPIA
T A NRZUASSIA
N IA
GEOR GIA
R OM A N IA
A Z ER
B AAIJA
N IQU E
M OZ
MB
PA KIST
A R GEN
T IN AA N
T
A NA
NA
ICJIKIST
A R A GU
E
IN D ON
V IET N A M Z IM B A B W
PHILIPPIN
ESYESIA
N EPA
L
KEN
A
U GA N D A
N IGER IA
B EN IN
KY R GY Z ST A N
r = 0.02
0.2
20
40
60
80
100
Global Integrity Index 2006: "De Facto and De Jure" Components of
Anticorruption Institutions
Source: Center of Public Integrity - http://www.globalintegrity.org/, and World Economic Forum..
24
Disaggregate and Aggregate Indicators
• Aggregate indicators have clear drawbacks (but not alone…)
• Many individual measures of governance provide imperfect
signals about broader concepts
– major advantage of aggregation is that allows reduction
in and transparent quantification of margins of error
• Aggregate indicators are more informative about broad
concepts, but at a cost of specificity
– average information from distinct sources (WGI, TI)
– contrast with “single-source” aggregates averaging
answers from same respondent (e.g. CPIA, GII, DB)
• here correlated errors are a major concern
• Not an “either/or” situation: aggregate indicators can be
disaggregated
26
Precision and Number of Sources:
Control of Corruption, 2006
Standard Error of
Governance Estimate
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0
3
6
9
12
Number of Data Sources
15
18
27
Source for data: 'Governance Matters VI: Governance Indicators for 1996-2006’, D. Kaufmann, A. Kraay and M. Mastruzzi, (www.govindicators.org);
A list of 10 suggestions on ways forward...
1. Many valuable existing ‘baths’, with clean (‘signal’)
& some dirty (‘noise’) water. Use & Improve
2. Since we deal w/ proxies: recognize & disclose
margins of error in all indicators.
3. ‘Trapping’ Informality (incl. state capture…): let us
not think that we solely rely on formal rules
4. Continue to develop “action-worthy” indicators,
complementing them with “outcome” indicators
5. Yet, more ‘policy circumspection’: ‘cats &
mousetraps’ vs. ‘injections’ -- what Matters is not
always ‘WB actionable’
6. Triangulate: do not rely exclusively on any one
indicator or source w/out checks/balances
28
Cont… w/ others
On Moving forward, cont.
7. Match indicator(s)/tool with desired objective: indepth country diagnostic, vs. broad country
coverage, and exploit complementarities (aggregate
& diagnostics; rules & outcomes)
8. Distinguish between 1st vs 2nd order problems and
prioritize -- Improving questions, sampling, testing
& correcting for biases important (vs. ideology,
correlated errors, ‘halo’ effects…?)
9. Perceptions do Matter, & likewise accounting for
reports from citizens and entrepreneurs – though
margins of error, yet not unique (& judgments galore)
10. Fully disclose data sets (& methods, comparability
across countries, periodicity, etc), & subject to
rigorous referee review -- & avoid censorship from
29
publication of data
Expressions...
Aristotle, Lord Kelvin, Einstein… already covered,
so onto the simple bumper sticker nowadays…
“if you think there are problems with the data,
try without it…”
Lets move forward together…, with some
circumspection, cats and mousetraps…
30