Transcript Do Accountability and Voucher Threats Improve Low

Do Accountability and Voucher
Threats Improve Low-Performing
Schools?
David N. Figlio and Cecilia Elena Rouse
NBER Working Paper No. 11597
August 2005
JEL No. I20, I21
Aim of the paper
Study the effects of the threat of vouchers
and school stigma in Florida on the
performance of low-performing schools
General Framework 1/3
• Florida’s program utilizes stigma by grading schools “A”
(excellent) through “F” (failing)
• Students in “F” schools are eligible for taxpayer-founded
vouchers (Opportunity Scholarship Program) cheques
gave to low income families in order to help them in the
school choice. The families can choose among private or
higher-rated schools
• OSP has two rationales:
– Fairness to reduce social stratification
– more competition into the provision of education (in fact, based
on the economic theory, an increased competition should force
schools to improve)
General Framework 2/3
• 1996 Florida began rating schools based on their
aggregate norm –referenced test performance
introduction of the Critically Low Performing Schools
List (four school rating groups)
• 1998 introduction of the new Florida Comprehensive
Assessment Test (FCAT) based on the SSS in grades 4
reading ,5 math, 8 and10 both in reading and math
• 1999 introduction of the A+Plan. It divided school in five
rating groups and it establish private school vouchers
assigned in event of chronic low performance. Two types
of tests: nationally norm referenced test  FCAT-NRT
and the curriculum based assessmentFCAT-SSS
the two tests assess different set of skills. The FCAT-SSS
tests a narrower set of skills but in greater depth than do
the broader nationally NRT
General Framework: A + Plan 3/3
• In the early years of the A+ Plan, schools were
identified as low performing:
• if fewer than 60% attain level 2 in reading, fewer
than 60% attain level 2 in math, and fewer than
50% attain level 3 in writing.
• In addition, schools that improved by more than
one grade level or retained an “A” were awarded
an additional $90-$100 per student.
Evidence of test score improvements
0.3
s2
Improvements in students learning 1/3
• Real gains or statistical artifact?
• If real gains due to : voucher threats or
other elements of the accountability
system such as grading stigma?
There are several alternative hypotheses
that might explain real gains:
1) mean-reverting measurement error
2) changing student characteristics
3) opportunistic teachers’ behaviours
Improvements in students learning 2/3
1) Mean – reverting measurement error
• The gains obtained by “F” rated-schools were due to mean
reversion. Indeed the gains obtained by these low rated
schools in the following year had been larger than the
average gains. This could be due to the “floor effect”
and/or “ceiling effect”:
Schools that scored highly on the test in one year are
disproportionately likely to experience lower than average gains the
following year
higher score obtained
in one year
lower than average gains
in the following year
• The school rating is based on test’s result of only one year, thereby it’s
possible that many of the F-rated schools had only a transitory low
grade due to low test scores. Their score would have increased in
subsequent years even in absence of the A+ Plan
Improvements in students learning 3/3
2) Changing students’characteristics
districts attempted to redraw school
attendance area boundaries in order to
improve the student characteristics of low
rated-schools.
3) Opportunistic schools’ behaviours
schools only focus on high-stakes
grades test, teaching to the students the
test’s matter.
Empirical framework 1/2
T ist is student i’s test score in school s in year t
F is dummy variable indicating whether or not the school attended by student i
received a failing grade of “F” in 1999.
YEARt is a vector of year effects
POSTt dummy variable indicating if the year is after the implementation of the
A+ Plan (in these data it is equal to one if the year is 2000 and zero otherwise)
GRADEist is a vector of dummy variables indicating the student’s grade in school
f s is a vector of school-fixed effects
m s school-level time trends predicted from the pre-1999 data
e ist is a normally distributed error term
 b key coefficient of interest. It measures the change in a student’s test score
resulting from her school having received an “F.”
Empirical framework 2/2
• Utilizing a difference-in-differences framework
• The Tist –Tist-1 is the outcome observed for two groups of test score
in school s for two time periods. One of the groups is exposed to a
treatment (A+Plan) in the second period but not in the first period.
The second group is not exposed to the treatment during either
period.
• The dummy variable POSTt captures possible differences between
the treatment and control groups prior to the policy change. F is
captures aggregate factors that would cause changes in outcome
even in the absence of a policy change. b multiplies the interaction
term, F is and POSTt
• This particular specification of the education production function
allows us to control for the student’s prior academic achievement (by
controlling for Tist-1) and this allows to have the right estimates of
the effect of the A+ Plan.
Data
• Student-level data from a set of participating school districts from
1995-2000.
• These data include scores on:
– FCAT-SSS Florida Comprehensive Assessment Sunshine State
Standards
– Norm-referenced NRT
– Basic student demographic attributes, including information on
student race, ethnicity, poverty status, limited English proficiency
status, and disability status.
• Data on third, fourth, and fifth grade NRT scores and FCAT-SSS
scores for grade 4 in reading and grade 5 in math. These data
include over 1500 schools
• Focus on elementary grades (from 1 grade to 6 grade) because they
comprised the vast majority of “F” schools
• Dependent variable is the normal curve equivalent of the test scores
• The mean is 50, the individual-level standard deviation is 21.06
Dependent variable 
Dummy
variables
0.07 s
Ss at 10%
Ability
•Study changes in academic performance that resulted from the earlier
policy of placing schools on a critically low performing list
Conclusion1/2
• Overall, they find that students showed significant improvement on
the high-stakes exam (the FCAT-SSS) in both reading and math.
• However, once they examine the effect of the accountability system
on a nationally norm referenced test and control for student
characteristics (by including the lagged test score), they find that
there was no large-scale relative improvement in average NRT
reading scores due to the A+ Plan among students in the lowest
performing schools.
• There were modest relative improvement in average NRT math
scores, although this improvement appears largely concentrated in
the high stakes grade.
• And while they find some evidence that sanctioned schools put
additional resources into low-performing students in math, more
generally they find little evidence that these schools treated students
differentially than low-performing schools that did not receive an F.
Conclusion 2/2
• A+plan is seen from Florida as a good tool to improve
the low performing schools in this way all children can
have a good quality of education.
• Florida with the institution of this provision is sure that
the improvement of the low performing schools conducts
to an improvement in children’ learning. This would be
logical
• But as the authors show the low performing schools in
order to avoid the stigma to receive an F (that is the
main reason why the school improve and not for the
voucher threat) they behave in an opportunistic manner
for example focusing on the low performing children and
leave aside the other subgroups of students, teaching
the high-stake test
contradiction with the two rationales of the OSP: fairness
and competition