Impact Evaluation for Land Property Rights Reform

Jonathan Conning Hunter College and The Graduate Center City University of New York 1

2

A brief polemic to make some obvious points

 Randomized controlled trials are not the gold standard   Best ideal tool in the toolkit for measuring ATE but does not trump other evidence.

The purpose is not to understand “

whether

projects work” but rather “

why and in what contexts projects can be expected to work

(Deaton,2009).”  Average treatment effect from a RCT by itself may be of limited value in guiding policy on specific subgroups.

  Example: ATE finds that project to build rural land registries had no impact. ○ should we never again build a land registry? ○ ○ are land registries in specific localities (say a land-scarce region) are not worthwhile? Have we learned enough about why project failed to re-design and try again? We can only make such judgments by bringing additional evidence, and our understanding of the mechanisms at work, to play. 3

The purpose

    is to improve understanding of the mechanisms that make projects work or fail to work.  ATEs measured in impact evaluation are useful and important, but..

 more broadly it is the process of building impact assessment into projects that generates knowledge for framing and focusing better decision making.

 The dialogue with policy makers and stakeholders about intended impacts and mechanisms and potential winners and losers, the challenges of data collection ….

Especially in “land property rights reforms” where interventions are deeper and more complicated (than, say, giving out iodine tablets) and success or failure often depends on context. Collect data to understand context and the mechanisms.

Don’t throw away data b/c can’t analyze it with sufficient ‘purity’.

4

Why do impact evaluation?

 If built into program implementation and it engages stakeholders and local governance structures it can offer useful framing and feedback for better understanding of mechanisms for better policy design.  Scaling up and replication  Accountability, open debate and good PR 5

Types of Intervention

 Land Administration, titling and other tenure reforms   Area and community-wide: cadastral surveys, improved registries and adjudication. Village-level boundaries and governance.

Individual: regularization, registration, issuance of ‘titles’.

 Redistributive Land Reforms   Imposed Negotiated  via Restitution 6

Few Impact Studies

 Very few carefully identified impact studies on land issues.

 Very few No purposefully designed impact studies with baseline.

 ○ Most existing studies had to make up for lack of data on comparison group and no baseline survey.  Evidence to date from natural experiments and observational data Feder et al in Thailand - QR, HH FE Galiani et al in Argentina -QR, ITT Field et al in Peru –QR, pipeline Carter and Olinto in Paraguay –structural model of selection Akee on Native-American lands in South-west, QR ○ Data collection should be built into program design. Much has been learned from evidence collected, concerns over biases in the estimates of ATE may remain in doubt.

7

 Property rights established via •

Individual actions

(e.g. investments to establish possession •

Community sanction

(e.g. customary tenure, neighbors acceptance) •

State Enforcement

(e.g. enforcement of legal title by courts or police)  Claims can conflict and overlap. Many disputes are settled privately, others politically.

 Land administration and reform programs aim to facilitate process and/or to politically re-assign rights.

 Implication for Impact Evaluation: • Program placement and timing non-random (politically determined) • Self-selection of beneficiaries • Differentiated impacts 8

Dar es Salaam, Tanzania

Source: Google Earth 2-9

Chogo village, Handeni, Tanzania

10

Household level Wider society/economy 11

Possible Positive Impacts

 Investment Demand Effect  Credit Supply Response  Gains-to-trade effects  A few others mentioned in literature:  Attitudes and outlook, civic participation, willingness to pay taxes, household labor supply, investment in education, area crime.

12

From: Pagiola (1999) adapted from Feder and Nishio (1999).

13

The endogeneity Problem.

How did the potential beneficiary get to the front of the line?

Negative Impacts?

       Badly designed top-down intervention could disrupt existing informal property arrangements (built upon Coasian bargains and balanced overlapping claims. Costly conflicts, rent-seeking Land grabs and inefficient rush to privatize common areas.

Strengthen elites while leading to loss of security for weakest stakeholders World Bank programs are supposed to be designed to avoid the above, but ‘the road to hell is paved with good intentions.’ All the more reason to study and use impact assessments, as well as case studies and participatory appraisals to understand mechanisms in small pilots before scaling up and to embed impact assessment in policy design process.

Even in the best designed programs, rarely is the intervention win-win. Should identify losers as well as winners.

15

Household Level Impacts

      Income, Production, Consumption, Safety Net Investments (under- or over-) Health, Education Market participation  Land Values, sale and lease activity  Labor supply  Credit Access Intra-household bargaining Attitudes and outlook  Measurement issues 16

Wider Impacts

 Reduction of conflict and transaction costs  Benefits or costs spillover to non-target groups  More active land markets  Labor markets  Financial Markets  Public finance and public goods  Environment conservation  Spillover effects may complicate evaluation 17

Data Collection

(See also Deininger (2004)) .

 .

  Baseline Build on standard LSMS-type surveys with added land module Collect information on:       land transaction histories, info on partners.

plot level data intra-HH distribution of rights and transferability property-related transaction costs, HH knowledge of law HH expected impact of reforms (e.g. expect gain in land value).

HH attitudes      Data from communities and reform programs. existing property rights arrangements, governance structures variables explaining program placement and beneficiary selection program rules and implementation features measures of community-level property rights insecurity 18

 Measuring treatment and outcome  Finding right comparison group  Interpreting findings 19

Finding a Comparison Group

 Comparison simple if treatment (e.g title) randomly assigned, but  Programs are targeted to specific groups and/or demand driven  e.g. most politically organized communities  Participants self-select  e.g. occupants have already made costly investments/taken chances, first in line value title more highly 20

Attribution Problem

 Participants might have had higher outcome even in absence of treatment.

 Might falsely attribute outcome to treatment when (fully/partly) due to failure to control for observable and unobservable differences between treatment and comparison groups  e.g. Participants more risk-taking, higher entrepreneurial drive, better political connections, better access to credit, etc.

21

Solutions

 Collect good baseline data  Data before and after intervention.

 Good outcome measures and proxies and data that may account for participation.

 Build comparison groups carefully   Via various randomization devices where possible By controlling for observable differences  Not difficult or costly if built into program design.

22

Solutions

 Compare different outcome of treated to that of comparison using:   D: Difference land value of titled before and after intervention  land value of titled vs. non-titled after intervention   DD: Double Difference

increase

in land value of titled vs. non-titled  Extensions: Differentiated/non-additive impacts, fixed effects (household, community) 23

24

25

26

27

Solutions

 Compare different outcome of treated to that of comparison using instrumental variables method:  Compare land value of titled vs. non-titled after intervention  Use program eligibility rules as instrument for selection into treatment  Extensions: Differentiated/non-additive impacts, fixed effects 28

Data Collection

(See also Deininger (2004)) .

  .

 Baseline Build on standard LSMS-type surveys with added land module Collect information on:      land transaction histories, info on partners.

plot level data intra-HH distribution of rights and transferability property-related transaction costs, HH knowledge of law HH expected impact of reforms (e.g. expect gain in land value).

     Data from communities and reform programs. existing property rights arrangements, governance structures variables explaining program placement and beneficiary selection program rules and implementation features measures of community-level property rights insecurity 29

30

Methods for building Comparison Groups

 Experimental Randomization  Pseudo-Experimental  Controlling for observable differences 31

Experimental Randomization

 Ideal  Often not feasible 32

 Pipeline e.g. comparison group from those still on waitlist (if order is not affected by group’s characteristics)  Geography e.g. treated if on one side of arbitrary boundary and unexpected  Program Eligibility Criteria Rule e.g. arbitrary eligibility cutoffs 33

34

Controlling for Observable Differences

 Regression methods  Propensity score methods 35