Transcript Document

Moving Along the Impact Pathway:
The Case of IAA in Bangladesh
John Antle & Roberto Valdivia
Oregon State University
Charles Crissman & Khondker Murshed-e-Jahan
WorldFish Center
Presented at Assessing the impacts of international agricultural research on poverty and
under-nutrition: A mid-term workshop for studies commissioned by the CGIAR Standing
Panel on Impact Assessment (2011 – 2013), London International Development Centre,
May 8-9 2012.
Project Goals
Estimate adoption, poverty reduction and household nutrition impacts
from the promotion of integrated aquaculture-agriculture technologies in
Bangladesh.
H1: DSAP recommended practices are economically feasible for more than 50
percent of the target population
H2: Incomplete adoption is explained by: (a) average productivity and/or cost of
production; (b) high variability in productivity and/or cost of production.
H3: Adopters have lower poverty and better nutrition than non-adopters.
H4: Impacts are the same for small and large farms.
H5: The TOA-MD model predicts adoption rates sufficiently well for use in ex ante
and ex post impact assessment.
H6: The TOA-MD model predicts aggregate impacts accurately without knowing
adoption-outcome correlations.
H7: Impacts within adopter and non-adopter sub-populations can be predicted
accurately without knowing adoption-outcome correlations.
Development of Sustainable
Aquaculture Project (DSAP)
• DSAP – objective to sustainably increase productivity through
Integrated Aquaculture Agriculture (IAA) which focused on
resource use efficiency through better utilization of resource
flows between farm enterprises
• Project utilized a strategy of decentralized local-level long-term
training, exposing farmers to a basket of 19 technologies and
management practices.
• Large scale project implemented in 34 of the 64 districts in
Bangladesh
DSAP Impact Assessment Data
• DSAP monitoring activity focused on
farm survey and regular monitoring
data from 260 participating farm
households from four districts
• Baseline in 2003/2004
• Training and extension during
repeated visits from 2003/2004
through 2005/2006 during which
regular follow up data was collected
via a series of whole farm
monitoring surveys
• Control: 123 non-project farmers
from the same districts were
surveyed in 2003/2004
PANCHAGARH
THAKURGAON
NILPHAMARILALMONIRHAT
KURIGRAM
RANGPUR
DINAJPUR
GAIBANDHA
JOYPURHAT
SHERPUR
SUNAMGANJ
NAOGAON
JAMALPUR
SYLHET
NETRAKONA
BOGRA
NAWABGANJ
MYMENSINGH
MAULVIBAZAR
RAJSHAHI
SIRAJGANJ
NATORE
KISHOREGANJ
TANGAIL
HABIGANJ
GAZIPUR
PABNA
NARSINGDI
BRAHAMANBARIA
KUSHTIA
MANIKGANJ
MEHERPUR
RAJBARI
DHAKA
NARAYANGANJ
CHUADANGA
FARIDPUR
JHENAIDAH
MUNSHIGANJ
MAGURA
COMILLA
SHARIATPUR
MADARIPUR
CHANDPUR
KHAGRACHHARI
NARAIL
GOPALGANJ
JESSORE
BARISAL BARISAL
BARISAL
LAKSHMIPUR
BARISAL
BARISAL
NOAKHALI
SATKHIRA
KHULNA
PIROJPUR
JHALOKATI
JHALOKATI BARISAL
BAGERHAT
FENI
RANGAMATI
CHITTAGONG
BHOLA
CHITTAGONG
PIROJPUR
PATUAKHALIPATUAKHALI
PATUAKHALI
BARGUNA
BHOLA
BAGERHAT BARGUNA
BARGUNA
PATUAKHALI
KHULNA
SATKHIRASATKHIRA
BAGERHAT
SATKHIRA
BAGERHAT
KHULNA
PATUAKHALI
PATUAKHALI PATUAKHALI
PATUAKHALI
NOAKHALI
BHOLA
BANDARBAN
COX'S BAZAR
DSAP Survey Data
Baseline
Project
Follow up
2002/03
2003/04
2004/05
2005/06
2011/2012
225
225
225
225
225
Secondary
Adopter
Control
225
123
123
123
123
Baseline covered 260 farm households – 35 were large commercial
rice-fish operations that are dropped for this analysis
Secondary adopter – selected from matching project village
Follow up survey will cover same households as in original project
123
Multi-Dimensional IA: Motivation
“Moving along the impact pathway”
• how can the research community respond to
stakeholders’ desire to understand tradeoffs and
synergies between economic, environmental and social
dimensions of sustainability?
• Given realities of data quality, costs of data and human
resources, how good is good enough?
• But…complexity does not trump principle of parsimony
Simulation-based IA: Motivation
• There is a need for a feasible, generic, transparent approach to multidimensional IA that can be implemented at “low cost” in terms of
data, time and human resources, that is based on received economic
and statistical theory.
• The simulation-based approach to IA responds to this demand, by
building on and integrating established concepts in the technology
adoption, statistics and econometrics literatures.
– It is a way to integrate various kinds of data to simulate the
economically feasible adoption rate and various indicators based
on quantifiable outcome variables
– It can be linked to econometric behavioral models and market
equilibrium models for aggregation and disaggregation
Some features of the simulation approach implemented in
TOA-MD:
• Is a parsimonious, transparent model: results can be easily interpreted in
relation to underlying data and parameters
• Can utilize all types of available data:
– survey data, experimental data, modeled data, meta data, expert data
• Provides a framework to carry out sensitivity analysis
– Can use preliminary or “minimum data,” provide guidance for efficient collection of
additional data when needed
• Can be used for prospective and policy-relevant IA (extrapolation; ex
ante; analysis of policy impacts)
– Overcomes the critical “support” assumption in the econometric approach that both
“treated” and “untreated” individuals are observed
• Estimates various kinds of impacts
– Conventional average “treatment” effects
– Mean and threshold impacts on adopters and non-adopters, at any degree of
adoption
– Policy-relevant impacts: taxes, subsidies, Payments for Ecosystem Services, etc.
Conceptual Model of Technology Adoption
and Impact Assessment (Antle 2011 AJAE)
mk
Technology adoption,
environmental econ, &
policy evaluation literatures
k
Contours of equal density
of joint distributions of ω
and outcome variable k, for
systems 1 and 2
2
mk(2,0)
mk(0)
mk(2)
l
l
mk(1)
mk(1,0)
l
l
1
0
r(2,0)
100
0
Impact indicators for
outcome k (in this case,
mean outcomes)
w = opportunity cost
Adoption curve for system 2
100
r(2)
Step 1: System choice
(ω)
Opportunity cost, system
choice and adoption
Opportunity cost w = v1 – v2 follows
distribution (w)
Generalize w to any ordering, e.g.
“willingness to adopt”?
Key point: a model of a
population using a
system, not a farm
using a technology
System 2: w < 0
(adopters)
0
System 1: w > 0
(non-adopters)
w opportunity cost
Map of a
heterogeneous
region
Opportunity cost distribution
• Key point: a fundamental piece of information about a
system is its economic feasibility: an upper bound on
potential use or adoption
• w = v1 – v2 is distributed in the farm population
–
–
–
–
–
–
“Every farm has its w”
mw = m1 - m2
w2 = 12 + 22 - 21212
Ex post: observe system 2, approximate system 1 (counterfactual)
Ex ante: observe system 1, approximate system 2 (counterfactual)
How to estimate 12?
• “Unobserved heterogeneity”: how to approximate?
• Populations can be stratified by various criteria: geographic,
technological and socio-economic
w
Derivation of adoption rate
from spatial distribution of
opportunity cost with
adoption threshold a = 0
This model shows the relationship between
the mean and variance of ω and the
economically feasible adoption rate. If the
mean of ω is positive (negative), the
adoption rate is less than (greater than) 50%.
Similarly, changes in variance have
predictable effects on the adoption rate.
ω>0
(w)
r(2)
50
ω<0
100
r (2,0)
Link to models with “selection on
unobservables”: ω is unobservable, but
observed or otherwise approximated
distribution of returns can be used to
estimate the moments of the ω distribution
Step 2: Impact Assessment
Adopter, non-adopter, and population means of
unconditional and conditional outcome distributions
mk
k
Key concepts: unconditional
outcome distributions
(ellipsoids) and outcome
distributions conditional on
adoption (filled). The latter
are truncated by adoption
decisions.
Adopter mean
2
Population mean
l
mk(0)
mk(2)
l
mk(1)
l
l
1
Non-adopter mean
0
r(2,0)
100
0
mk(h) = mean of outcome k
when all farms use system h
mk(0) = population mean at
predicted adoption r(2,0) for
adoption threshold a=0
mk is mean of
outcome k
When k is expected
returns, the population
mean is maximized at
the adoption rate r(2,0)
w
100
r(2)
Counterfactual mean of adopters and the
Average Treatment effect on the Treated (ATT)
mk
Similarly, the distribution of
system 2 for ω> 0 can be
used to construct the
counterfactual for nonadopters and the average
treatment effect on the
untreated (ATU).
k
2
mk(2,0)
mk(0)
mk(2)
l
l
mk(1)
mk(1,0)
l
l
1
0
r(2,0)
100
0
Counterfactual mean of
adopters
ATT at r(2,0)
Counterfactual mean of nonadopters
ATU at r(2,0)
100
r(2)
w
Means, counterfactuals, ATT, ATU, ATE, LATE, MTE
(see Heckman, Urzua and Vytlacil, RE Stat 2006)
Given the joint distributions of adoption variable ω and outcome k, we can compute
all treatment effects as well as mean and threshold indicators at all adoption rates
simulated by varying the adoption threshold a.
Also we can show that:
ATE = mk(2) - mk(1) = r(2,a) * ATT/100 + {100 – r(2,a)}*ATU/100
MTE = dmk(a)/dr(2,a)  LATE/r(2,a)
mk
ATT = average treatment
effect on the treated
(adopters)
Adopter mean
Population mean mk(a)
MTE
mk(2)
ATE
Adopter CF
mk(1)
0
Non-adopter mean
0
r(2,a)
r(2,0)
100
ATU = average treatment
effect on the untreated
(non-adopters)
100
Non-adopter CF
Effects of selection on impact indicators:
Sorting gain and counterfactual bias
mk
k
2
mk(2,0)
mk(0)
mk(2)
l
l
mk(1)
mk(1,0)
l
l
1
0
Degree of selection bias in
counterfactual depends on
correlation between outcome
and adoption variables. If
correlation is zero, means and
counterfactuals have zero slopes,
and population mean is linear
with slope equal to ATE.
r(2,0)
100
0
100
r(2)
w
Threshold Indicators
The same concepts can be used to define and simulate threshold indicators
(poverty rates, poverty gap, environmental risk, nutritional risk, etc).
Threshold indicator for system 1
(areas c+d+e+f)*
k
Threshold indicator for system 2
(areas b+d)
b
Threshold indicator for system 1
(areas e+f)
b
System 1
mk(2,0) 
c
k
d
e
System 1
mk(2,0) 
f
c
d
e


mk(1,0)
f


mk(1,0)
System 2
System 2
w
0
BEFORE ADOPTION OF SYSTEM 2
* Ignoring areas outside the contour as negligible
w
0
AFTER ADOPTION OF SYSTEM 2
Policy-Relevant Impacts and “Local” Impacts: E.g., a policy reducing
constraints on adoption to increase adoption rate from rc to r(2,0)
mk
k
2
l
l
l
l
1
0
rC
As in Heckman and Vytlacil
(Econometrica 2005) we
consider policies that affect
adoption but do not change the
unconditional distributions of k
and ω.
r(2,0)
100
0
100
r(2)
w
Implications for IA
• There is a fundamental symmetry between “ex ante” and “ex post” IA:
each involves parameterization of:
– the unconditional joint distributions between w and the outcome variables
– the mechanism or process determining choice between systems
• By characterizing the joint distributions of the adoption variable w and the
outcome variables, we can simulate the economically feasible adoption
rate of system 2, and all relevant average and threshold impact indicators
and “treatment effects.”
• Data collection should focus on characterizing these joint distributions in
the relevant populations.
• Characterization of counterfactuals (both ex post and ex ante) should
utilize all relevant information: primary, secondary, experimental,
modeled, expert, meta data.
Counterfactual system design
• In most cases, counterfactual system can be constructed as a
transformation of the observed system
– E.g., changing crop variety leaves most of the observed system intact, but may alter
productivity and land allocation
– E.g., introduction of IAA changes management , productivity and “bio-resource flows”
but not the components of the system
• Random coefficient model
– In general, v2 = v1 + , where  is a convolution of v1 and v2
– Then v2 = (1+ /v1) v1 = riv1 giving r = (1+ /v1) where r = mr + r,   (0,1)
– Using this model, with observations on one system and plausible bounds on mr & r we
can approximate mean, variance and between-system correlations for the other system
– data for mr & r can come from various sources: observations, models, experiments,
meta data
– Draft paper available on this concept
Bangladesh DSAP Case Study
See Jahan and co-authors, Aquaculture Research (2010),
Agricultural Systems (2011)
“The impact of long-term IAA training provided to small-scale farmers in Bangladesh is
assessed using panel data from 260 project and 126 control farmers who were monitored
from 2002/2003 to 2005/2006. We find that the training had a significant positive impact
on farmers’ technical efficiency, total factor productivity and net incomes. These result in
higher food consumption and better nutrition for trained households compared to control
farmers.”
• We interpret these results as “ATE”
Bangladesh DSAP Case Study (2)
• Here we replicate and extend Jahan et al. using TOA-MD
• System 1: Farms with no training support and low integration of
aquaculture and agriculture. Low integration is 2 or fewer managed
bio-resource flows among farm enterprises in 2002-2003
– control group data show no significant trend from 2002/03 to
2005/06
• System 2: Farms with training support and highly integrated
aquaculture agriculture (treated group in 2005/06)
• Farms stratified by small and large (small is less than 1 hectare)
DSAP Data
• 260 participating farms in the baseline survey:
– 59 classified as System 1
– after the training and extension activities, 46 of these switched to
System 2, implying an adoption rate of about 78%
• Farm size, pond size, family size, and non-farm income
• Production activities defined as rice, vegetables, other crops, poultry
and livestock, and fish
• Net return calculations based on per-farm revenues and costs of each
activity
Impact indicators
• Mean farm income and per/capita income - $1500/year and
$356/year. Fish culture contributes about 16% of farm income and
11% of total annual income.
• Poverty rate – an estimated 45% to 50% are live below the $1.25/day
poverty line – in the survey 54% are below the poverty line
• Food consumption – the national Household Income and Expenditure
Survey shows that on average rural households consume 2253 kilo
calories of which fish contribute 52 kilocalories.
The TOA-MD Model
• Software with documentation in SAS and Excel, available to
registered users at tradeoffs.oregonstate.edu
– Self-guided course and training workshops
• Represents heterogeneous populations with multiple strata
(e.g., small, large farms; agro-ecozones; etc)
• Generic whole-farm structure
– Crop, livestock and aquaculture sub-systems with multiple
activities within each system
– Farm household size, non-ag income
– Income, poverty and generic indicators (mean, threshold)
• Technology adoption/impact; ecosystem service supply;
environmental change & adaptation
The TOA-MD Model
• Some features
– Parameter parsimony: fundamental parameters are:
• 2 means, 2 variances and 1 between-system correlation of system expected returns (5
parms)
• 2 means, 2 variances, 3 correlations for each outcome variable (7 parms x N
outcomes)
– Unconditional joint outcome distributions for each stratum of the
population is normal (thus, aggregate distribution is non-normal)
– Predicted “adoption rate” (choice between systems) is based on
expected returns over a relevant decision period
– When system 1 and 2 data are not matched, the between-system
correlations cannot be estimated from observations, so plausible values
are estimated and used in sensitivity analysis.
• Show the model…
Adoption curves for small and large farms
1000
800
Observed adoption rate was
approximately 76% based on
59 observations , very close
to the predicted adoption
rate of 78% averaged over
small and large farms.
600
400
200
Small Farms
0
0
10
20
30
40
50
60
70
80
90
100
-200
-400
-600
-800
-1000
Mean opportunity cost occurs at 50%
adoption rate in this model. If it is
negative then the adoption rate must
exceed 50%, as in this case. The
predicted adoption rate depends on the
mean and the variance. Sensitivity to
model parameters can be performed
easily.
Large Farms
Mean net returns/farm for small farms
900
700
ATE = 144
TT = 205
500
Adopter Mean
Adopter CF
300
Pop Mean
Non-adopter mean
100
Non-adopter CF
0
-100
-300
-500
10
20
30
40
50
60
70
80
90
100
Small farm net returns: ATT, ATU and MTE
800
600
400
200
ATT
0
ATU
0
-200
-400
-600
-800
10
20
30
40
50
60
70
80
90
100
MTE
Poverty rates for small and large farms
100
90
80
70
60
50
40
100
Small Farm Adopter
30
Small Farm Adopter CF
ATE = -8.3
TT = -11.6
Large Farm Adopter
90
Large Farm Adopter CF
Small Farm Population
20
80
Small Farm Non-adopter
10
Large Farm Population
70
Small Farm Non-adopter CF
Large Farm Non-adopter
0
0
10
20
30
40
50
60
70
80
60
90
100
Large Farm Non-adopter CF
50
40
30
20
ATE = -4.6
TT = -8.1
10
0
0
10
20
30
40
50
60
70
80
90
100
Mean fish consumption in small farm
households (kcal/person/day)
100
90
80
ATE = 13.4
TT = 16.5
70
60
Adopter Mean
Adopter CF
50
Population Mean
Non-adopter Mean
40
Non-adopter CF
30
20
10
0
0
20
40
60
80
100
Percent of small farm households exceeding the
population average fish consumption of 51.6
kcal/person/day
100
Note that more than 50% of
adopters exceed average
consumption, whereas nonadopters are very low and
have very low incomes.
90
80
70
60
Adopter Threshold
Adopter CF
50
ATE = 30.3
TT = 38
Population
Threshold TU
Non-adopter CF
40
30
20
10
0
0
10
20
30
40
50
60
70
80
90
100
Large farm households exceeding the population
average fish consumption of 51.6 kcal/per/day (%)
100
Adopter Threshold
Adopter CF
90
Population
80
Non-adopter Threshold
Non-adopter CF
70
60
50
40
30
20
ATE = 33.2, TT = 34.3 (note they are close because correlation
between outcome and opp cost is near zero)
10
0
0
10
20
30
40
50
60
70
80
90
100
Sensitivity analysis to between-system
correlation RHO12, small farms
1000
900
Mean returns
800
Adoption curves
600
700
400
500
Adopter Mean RHO12=0.85
200
Adopter Mean RHO=0.95
RHO12=0.85
Pop Mean
300
Non-adopter mean RHO12=0.85
RHO=0.95
0
0
10
20
30
40
50
60
70
80
90
100
Non-adopter mean RHO12=0.95
RHO=0.6
Adopter mean RHO12=0.6
100
Non-adopter mean RHO12=0.6
-200
0
-400
10
20
30
40
50
60
70
80
90
100
-100
-600
-300
-800
-500
-1000
120
100
90
100
Mean fish consumption
80
70
Poverty rates
80
Adopter Mean RHO12=0.85
60
Adopters RHO=0.85
Adopter mean RHO12=0.95
Adopters RHO=0.95
Adopter mean RHO12=0.6
Population Mean RHO12=0.85
50
Adopters RHO=0.6
60
Population mean RHO12=0.95
Non-adopters RHO12=0.85
Population mean RHO12=0.6
Non-adopters RHO12=0.95
Non-adopter Mean RHO12=0.85
40
Non-adopters RHO12=0.6
Non-adopter mean RHO12=0.95
Non-adopter mean RHO12=0.6
40
30
20
20
10
0
0
0
10
20
30
40
50
60
70
80
90
100
0
10
20
30
40
50
60
70
80
90
100
Conclusions
• TOA-MD model predicts IAA adoption rate very close to observed rate
• IAA technology has substantial positive impacts on farm income,
nutrition -- to be further verified with additional new data
• Analysis shows IAA technology has substantial income and nutritional
benefits for both small and large farms.
• Selection MAY be (but is not necessarily) important in predicting income
and nutrition effects sufficiently accurately to draw meaningful policy
implications.
• Data quality is the greatest challenge to meaningful impact
assessment
– Especially for large surveys based on RECALL
• How “adoption” is defined is important – “adoption of technologies” or
“choice between systems”?
Conclusions
•
•
Work to be done:
– complete new surveys
– compare simulation and ex post statistical analysis
Extensions
– Further testing/validation: adoption, extrapolation
– Further explore systematic methods to combine primary, experimental,
modeled, expert, and meta data
• Using crop simulation models to construct counterfactuals for CC
impact & adaptation analysis
• Use for technology adoption
• Improve survey/field experiment design
– Develop methods to estimate standard errors
• Can use bootstrap to construct SEs but implementing in publicly
available software a challenge (try to maintain simplicity, generic
structure)
– Link to market models (Valdivia et al; IMPACT; other)