Transcript Overview: - NYU Steinhardt

Dana Burde
Visiting Assistant Professor
Department of Humanities and
Social Sciences
New York University
[email protected]
Panacea or Straitjacket?
A Qualitative Researcher Reflects on a
Randomized Trial in Afghanistan
Randomized Trial Assessing CommunityBased Schools in Afghanistan
Co-Investigator (for the large-scale randomized trial):
Leigh Linden, Assistant Professor, Economics Department and School of
International and Public Affairs, Columbia University
Agency studied: Catholic Relief Services
Research assistants: Matt Hoover, Dr. Saeed Mahmoodi, Nathan Falkner, Nafi
Olomi, Nicole Rigg, Amy Kapit.
Funders (from the inception of the study):
Columbia University Institute for Social and Economic Research & Policy
($15,000);
Weikart Family Foundation ($75,000);
United States Institute of Peace ($40,000);
Spencer Foundation ($40,000—pilot; $483,000—large scale study);
National Science Foundation ($100,000).
Outline of today’s talk
1.
2.
3.
Brief overview of arguments: Heated pro/con debate
between qualitative researchers and quantitative
researchers regarding the uses and effectiveness of
randomized trials
Study design: Description of the randomized trial of
community-based schools in Afghanistan and the
ways in which qualitative study components
complemented the quantitative design
Discussion: Design, execution, ethics, and research
culture—how these issues affected our randomized
trial
The Pro/Con Debate
Those in favor of randomized trials say they provide:
 Evidence-based findings that show more robust
results (than other kinds of studies)
 Answers to key questions in education research
This position is epitomized by this quotation:
“…[randomized controlled] experiments provide a
better warrant [for causal conclusions]…than any
other method. So if experiments can be conducted
in schools they should be. Not to use them requires
a very strong justification” (Cook, 2003, p. 117).
The Pro/Con Debate
Those who oppose randomized trials accuse them of:
 Weakness in the research design and execution of the study;
 Ethical shortcomings; and
 Promoting a culture of privileging quantitative methods
over qualitative.
This position is epitomized by this quotation:
“…there are formidable limits to the randomized control approach—limits
which the proponents of this methodology have not sufficiently
recognized. To achieve ‘rigor’ in education research, we need to adopt a
more realistic grasp of the problems that plague attempts to implement
research results as well as a greater openness to other approaches that
offer crucial insights into social processes in organizations. For these
purposes, qualitative methods, including participant observation and indepth interviews, are likely to be the most promising” (Lareau,
forthcoming, p. 146).
Argument
Based on our study in Afghanistan, I argue that:
 Randomized trials are excellent tools for assessing program
impact and are underused to study education programs
conducted by international NGOs
 Strong design and execution of place-based randomized
trials can overcome some weaknesses identified by critics
 Questions raised regarding ethics can be addressed and
should not preclude using randomized trials
 In agreement with the critics, I argue randomized trials
should not eclipse other forms of research; they are not
appropriate for every impact study; qualitative methods can
address their weaknesses and complement randomized trials
to great effect
Overview of Study
The study:

investigates whether community-based education programs
improve child welfare by comparing villages that receive
schools to villages that have not yet received them and by
comparing children in these villages.

examines attendance/enrollment, academic achievement, child
labor, and social benefits (protection, social networks).
Phases:

Qualitative pilot study: Tested field procedures 2005-2006

Randomized trial 2007-2008: Worked with a large international
US-based NGO, Catholic Relief Services (CRS), to randomly
assign their community-based schools to eligible villages (i.e.,
villages that had none previously)

Complementary qualitative case study of program
Program: Partnership for Advancing Community
Education in Afghanistan (PACE-A)

PACE-A is a five-year, $24 million USAID-funded
program:



meant to “expand quality learning and life opportunities for
marginalized communities and their children in Afghanistan”
(PACE Summary, 2006, p. 1)
Consortium of 4 NGOs tasked with providing hundreds of
community-based schools in 20 provinces across Afghanistan
The CRS portion of the program was intended to reach
approximately 100 villages by the end of 2008

Schools were to be phased in over several years
CRS Community-Based Schools
CRS selects districts and communities according to:
 Security; Availability of teachers; Level of community
interest; Community willingness to mobilize resources
(teachers’ salaries); Support from the Ministry of
Education (agreement); Community willingness to
provide space for the school (room in a home or
mosque).
 CRS provides training to teachers, materials for the
classroom (government curriculum), and regular
monitoring to track progress over time.
Study Hypotheses
Hypothesis 1 Attendance/Enrollment: Children living in a
village that receives a CRS school will be more likely to
enroll in school and will attend more regularly than children
in other villages.
Hypothesis 2 Learning: Children who live in villages that
receive a CRS school will perform better on academic
achievement tests on math and language skills than children
in the villages that do not receive a CRS school.
Hypothesis 3 Indirect Social Outcomes: Children who live in
villages that receive a CRS school will report higher levels
of social integration, higher levels of security, and higher
levels of safety than children in other villages.
Original Study Design
Yes - CBS in 2007
Group 1:
Normal CRS Class,
10 Villages, 500 HHs
Group 2:
Community education,
10 Villages, 500 HHs
Group 3:
Disciplinary Instruction,
10 Villages, 500 HHs
Group 4:
Community Education
and Disciplinary Instruction,
10 Villages, 500 HHs
No - CBS in 2007
(but yes for 2008)
Group 5
No CRS Schools,
40 Villages,
2,000 HHs
Actual Study Design
CRS community-based schools
N = 31 villages; N = 1,253 households;
N = 2,036 children aged 6-11
Yes in 2007
No in 2007
Group 1: Treatment
CRS Schools,
13 Villages,
637 HHs
Group 2: Control
No CRS Schools,
18 Villages,
616 HHs
Qualitative component
Case study of the CRS program to complement the RT and
learn more about:

program implementation;

place, i.e., country, province, districts, and villages in
which we are collecting data;

institutions that are part of our study or influence it: i.e.,
households, families, schools (government and CBS),
mosques, SMCs, village shuras, ministry of education.

individuals that are part of our study or influence it, i.e.,
children, parents/heads of households, teachers, arbobs,
CRS education staff members, CRS managers, PACE-A
directors
Map of Afghanistan
Ghor Province, Afghanistan
Ghor Province
Discussion: Design
Criticism: Addressing only narrow questions misses significant
variables in education
Our study: “Schools” constitute a significant variable
Criticism: Extent to which variables can be manipulated
accurately, e.g., networks
Our study: Intent was not to manipulate networks, but rather to
measure the impact of schools on networks, and other
secondary effects
Limitation: We can say that the program did not have an impact
on friendship networks, but need qualitative methods to say
why
Discussion: Execution
Criticism: Adherence to treatment; “contamination” of
control by treatment
Difficulties our study faced in execution:
• Persuading CRS to carry out a randomized trial
• Avoiding contamination
• Preserving the integrity of the randomization:
• Communication breakdowns (sample size decreased)
• Inclement weather (delayed start to survey)
• Security (attacks on staff, tribal warfare, etc. sample size
decreased to 31 villages)
• Hiring and training survey staff
Discussion: Ethics
Criticism: Unethical to deny treatment
Ethical issues our study faced:
 Sample selection: “Needs assessment” versus
randomized selection
 Temporary denial of treatment: Program was
intended to be phased in over time, regardless
 Program benefits not clear given the basic program
intervention
Discussion: Ethics
Additional ethical issues our study faced:
 Protection for the researchers was a far greater
problem than the protection of subjects
 IRB was silent on this point
 In fact, IRB protections for subjects put researchers
at risk:


Confidentiality
Signed documents
Discussion: Research Culture
A culture of preference for randomized trials pervades among some
research circles.
Qualitative methods are invaluable in addressing weaknesses of
randomized trials in several ways:
 A strong qualitative pilot study enhances the randomized study
design and increases the probability of successful program
intervention as well as successful execution of the study.
 Qualitative methods used in conjunction with a randomized trial
counteract the tendency of randomized trials to focus on narrow
cause and effect relationships.
 Complementary qualitative case study and ethnographic methods
provide more confidence in quantitative data analysis,
interpretation, and provide additional detail to explain findings
that randomized trials cannot.
Conclusion
Randomized trials -- crucial method to test multiple
empirical questions across the social sciences.
Critics have valid concerns -- randomized trials
should not be allowed to eclipse qualitative
methods.
As in all research, ethical questions raised by
randomized trials must be taken seriously.
Properly designed and executed, randomized trials
can produce robust and significant data even in
the most difficult circumstances.