Transcript Evaluating the Vermont Mathematics Initiative in a Value Added Context

Evaluating the Vermont
Mathematics Initiative (VMI)
in a Value Added Context
H. ‘Bud’ Meyers, Ph.D.
College of Education and Social Services
University of Vermont
Douglas Harris, Ph.D.
The Evaluation Center
The Vermont Institutes, Montpelier, VT
Purpose
 Describe the context, purpose, structure, staffing and content of the





instructional program for teachers
Describe the logic model that provides the framework for evaluation of
the program
Examine the methodology for assessing the value added to student
learning by teacher preparation in mathematics
Present findings from cross-sectional analyses of eight cohorts of
students who have been taught by VMI teachers and four cohorts of
control students.
Present findings from a qualitative evaluation of the process of
implementation of the components of the program from the
perspectives of participants and administrators
Present recommendations arising from the findings of the evaluation
Context, purpose, structure, staffing
 A field based masters degree in Education
with a primary emphasis on mathematics
content
 Partially funded by local districts, partially
MSP
 Focus on Grades k-8
 Includes focus on teacher leadership
 Includes action research
 Three years
 Mathematicians and educators
Content-12 Courses
Course 1: Mathematics as a Second Language (3
credits)
Course 2: Functions and Algebra for Elementary
Teachers (3 credits)
Course 3: Trigonometry for Elementary Teachers (2
credits) and Algebra and Geometry II (1 credit)
Course 4: Measurement, Geometry, and Probability
for Elementary Teachers (3 credits)
Course 5: Number Theory for Elementary Teachers
(3 credits)
Courses…contd.
 Course 6: Statistics, Action Research, and
Inquiry into Effective Practice, I (3 credits)
 Course 7: Statistics, Action Research, and
Inquiry into Effective Practice, II (3 credits)
 Course 8: Algebra and Geometry for
Elementary Teachers, III (3 credits)
 Course 9: Statistics, Action Research, and
Inquiry into Effective Practice, III (3 credits)
 Course 10: Calculus for Elementary
Teachers, I (3 credits)
Courses…contd.
 Course 11: Calculus for Elementary
Teachers, II (3 credits)
 Course 12: Capstone VMI Experience (3
credits)
Methods
 Quantitative- Cross-sectional and longitudinal

Two intervention groups




Group 1=16 schools, multiple VMI teachers
(HiEnd)
Group 2=22 schools, single teacher per school
(RegIntervention)
Control group=22 schools, matched to Groups 1
and 2 (combined) on demographics
Longitudinal comparisons for 2 cohorts, 1999
and 2000, grades 4 to 8. (See Figure 2)
Methods
 Qualitative
 Interviews







Participants
Graduates
Administrators
Observations
Review of course materials and participant
portfolios
Course evaluations
Interviews of School Staff and Leadership
Sample Characteristics
Figure 1
Comparison of Free/Red Lunch- Schools (%)
Percent Free/Red Lunch
40
35
30
25
2001 Free/Red Lunch- School (%)
20
15
10
2001 Free/Red
Lunch- School (%)
VMI
Control
32
36
Groups of Schools
Free or reduced price lunch eligibility is a metric for poverty used by the Vermont
Department of Education in the calculation of Adequate Yearly Progress. At grade
4 for each cohort when it was measured in the present study, it yields a statewide
average of about 34%. Both the VMI and Control schools were about at the state
average. Despite a 4 percent differential in poverty level between the VMI and
Control groups, comparisons of the VMI and Control groups of schools for both of
the cohorts indicated no significant differences among the matched schools with
respect to poverty levels as documented by the Vermont Department of Education
in 2001.
Longitudinal comparisons
Figure 2
Current and Potential Trend Studies with 3 data
points for VT Data
Others are cross-sectional, independent group
comparisons
Year
Gr 4
1999
x
2000
x
2001
x
Gr 5
Gr 6
Gr 7
Gr 8
Gr 9
Gr10
2002
2003
x
2004
x
x
2005
x
x
2006
x = Comparisons
x
x
Question 1: What do comparisons with comparable schools show, over time,
when students are matched from grade 4 through grade 8 and grade 10
during the two cohort groups (1999-2005, and 2000-2006)?
Figure 3
Scaled Scores
Intervention and Control Groups from 1999 to 2005 Matched
Pairs Scaled Scores
720
VMI
670
620
Control
1999
2003
2005
VMI
662
720
745
Control
657
716
734
Years of Intervention
Significant differences in 2005. t=3.22, p<.001, df 1, 1300.
Intervention and Control Groups from 2000 to 2006 Matched
Pairs Scaled Scores
Scaled Scores
Figure 4
720
VMI
670
620
Control
2000
2004
2006
VMI
671
720
728
Control
668
708
720
Years of Intervention
Significant differences in 2004. t = 5.27, p<.001, df 1, 1488
Significant differences in 2006. t = 3.39, p<.001, df 1, 1107
Discussion
As indicated in Figures 3 and 4, each of the VMI cohorts
outperforms the Control schools in similar patterns of
difference. Results for comparisons at the first data point for
each cohort (1999 and 2000) are not significantly different
while they emerge as significant in 2005 and 2006.
Question 2: How do levels of implementation of the VMI curriculum
in classrooms relate to levels of achievement of students
who experience them? (1999 Cohort)
Figure 5
Scaled Scores
Intervention and Control Groups from 1999 to 2005 Matched
Pairs Scaled Scores
750
725
Reg I
700
Control
675
HiEnd
650
1999
2003
2005
Reg I
650
708
735
Control
653
709
735
HiEnd
667
725
753
Years of Intervention
Significant differences in 2005. adjusted for differences in 1999,
HiEnd v Reg and Control, F = 7.96, df 2, 1235, p <.001
Question 2: How do levels of implementation of the VMI curriculum
in classrooms relate to levels of achievement of students
who experience them? (2000 Cohort)
Figure 6
Scaled Scores
Intervention and Control Groups from 2000 to 2006 Matched
Pairs Scaled Scores
750
725
Reg I
700
Control
675
HiEnd
650
2000
2004
2006
Reg I
664
710
722
Control
667
706
719
HiEnd
674
727
730
Years of Intervention
Significant differences in 2006 adjusted for differences in 2000,
HiEnd v Reg and Control, F = 9.91, df 2, 1093, p <.001
Note: The average standard deviation for groups is about 40 points across all groups.
Mean differences between groups is thus approximately .25 S.D. between HiEnd and Control groups.
Discussion:
VMI HiEnd Schools significantly out-performed
the matched schools and appear to carry most of
the difference between the VMI and matched
schools in both cohorts from 1999 through 2006.
Analyses of covariance adjusted the score
differences at the first two data points in each
cohort. The final data point for each cohort
represents group differences at grade 10.
Question 3. Are school level effects related to income level of students and
levels of content implementation of the VMI curriculum? (1999 Cohort) Do
gaps close?
Free Lunch vs Others Scaled Scores 1999 Cohort
ANCOVA
740
720
Scaled Scores
Figure 7
700
Reg I (E)
680
Control (E)
HiEnd (E)
660
Reg I (NE)
640
Control (NE)
620
HiEnd (NE)
600
580
1999
2003
2005
Reg I (E)
638
692
694
Control (E)
646
697
720
HiEnd (E)
650
699
725
Reg I (NE)
654
715
722
Control (NE)
656
714
721
HiEnd (NE)
670
729
727
Year
Question 3. Are school level effects related to income level of students and
levels of content implementation of the VMI curriculum? (2000 Cohort) Do
gaps close?
Free Lunch vs Others Scaled Scores 2000 Cohort
ANCOVA
740
720
Scaled Scores
Figure 8
700
Reg I (E)
680
Control (E)
HiEnd (E)
660
Reg I (NE)
640
Control (NE)
620
HiEnd (NE)
600
580
2000
2004
2006
Reg I (E)
635
695
723
Control (E)
639
700
722
HiEnd (E)
643
706
723
Reg I (NE)
652
713
727
Control (NE)
656
705
726
HiEnd (NE)
661
723
728
Year
Discussion:
The baseline, grade 4 performance is comparable for all groups. Not
surprising is the observation that whether in intervention schools or
matched schools, students eligible for free or reduced lunch score
significantly lower than their non-eligible for free or reduced price lunch
peers. This pattern continues through the grade 8 testing. Secondly, the
grade 10 Free-lunch eligible students are significantly outscored by their
non-eligible peers in the matched schools. No surprise there. But, rather
surprising is that given the lack of cohort differences at grades 4 and 8,
the free or reduced lunch eligible students are significantly out-scoring
their eligible peers at grade 10 and they are gaining on students who are
not eligible for free or reduced lunch in the matched schools. The mean
score for the group of VMI Free-lunch eligible students (723, in 2006) is
approaching that of the Matched Schools non-eligible students (726). In
other words, the gap is narrowing between the free or reduced lunch
eligible students and their non-eligible peers in the matched schools.
These results were not obtained for the regular VMI schools in 2005
(Figure 5) and we have no explanation for that result at this time.
Quantitative Summary
 Longitudinal comparisons of VMI and Control
schools indicate a persistent pattern of
advantage for VMI schools
 Longitudinal comparisons indicate that the
VMI schools, with more VMI teachers, had a
significantly greater long term gain on
percentile ranks. Stronger teachers may not
have as great an effect in the shorter term
when isolated.
 Longitudinal comparisons suggest that SES
differences are reduced over time when high
quality mathematics instruction is offered to
all students
Qualitative Findings (2005 and 2006)











Impact on Participants and Their Teaching
Finding 1: Mathematics Content
An overarching theme is the impact of the VMI experience on the teacher’s own
understanding of mathematics content. This is consistent among participants in all three
years of the VMI experience and remains the highest priority of graduates as they reflect on
the VMI experience
Finding 2: Increased Confidence Related to Mathematics
Increased understanding of mathematics content impacts the confidence of participants as
related to mathematics, to teaching mathematics, and to their enthusiasm for and enjoyment
of mathematics.
Finding 3: Impact on Instructional and Assessment Practice
Participants graduates (many of who are math teacher leaders), and principals report that
the instruction in VMI, increased content knowledge, and increased confidence have had
major impact on instructional and assessment practice in the classrooms of participants and
of teachers impacted by VMI-trained math teacher leaders
Finding 4: The Impact of Action Research on Classroom Practice
The impact of action research is mixed, with some participants, graduates, and principals
reporting considerable impact, others less so.
Finding 5: Principal Support
The active support of principals makes a profound difference in VMI participants’ work in the
classroom and in leadership positions.
More Qualitative Findings
















Finding 6: Personal Impact on Participants
Beyond the impact of VMI on teachers in relation to math content, instruction, and leadership, the program
also has profound personal impact on many participants.
Impact on Students
Finding 7: Transfer of VMI Content to the Classroom
Teachers report direct transfer of mathematics content used in VMI to the math experiences of their
students.
Finding 8: Impact on Student Problem Solving
Teachers and teacher leaders report that the problem solving emphasis in VMI has significant impact on
their understanding of how to engage students in problem solving in the classroom.
Finding 9: Impact of Action Research on Students
Some teachers believe that the interventions begun in their action research projects will continue to impact
their students over time.
Impact on Teacher Leadership in Schools and Districts
Finding 10: Impact on Teacher Leaders
Teachers working as teacher leaders credit VMI for providing the knowledge, confidence, and support for
them to take on leadership roles. This is true of teachers working in leadership at the school, district, and
state levels.
Finding 11: Impact on curriculum, instruction, and assessment
Teacher leaders report that the knowledge base and contacts developed during VMI have significant impact
on the development and implementation of mathematics curriculum, instruction, and assessment.
Finding 12: Need for Ongoing Professional Community
There is a common desire among VMI graduates to maintain the type of professional learning community
afforded them through VMI.
Recommendations
Recommendations Related to Continued Study of the Vermont Mathematics Initiative




Continue to gather longitudinal data from the State of Vermont’s Assessment
System. .
As Vermont transitions to statewide assessment utilizing the New England
Common Assessment, carefully analyze the logic and structure of the NECA and
review VMI course content in relation to the Grade Level Expectations upon
which this assessment is built.
Continue qualitative analysis utilizing existing data sources and consider adding
series of observations within VMI participants’ classrooms to better understand
what exactly is happening in those classrooms.
Select and implement a “theory into practice” change model considering, for
example:
o the IBM/Harvard School of Business Change Toolkit and/or
o the McREL Balanced Leadership Model.
o the CBAM model of school change
o the New Structure of School Improvement model (Joyce and Calhoun)
Educational or Scientific
Importance of the Study
The National Research Council, (2004) US Department
of Education (2004) and researchers at RAND (2004)
have cited the need for better evaluations of teacher
professional development programs. The
combination of mixed method approaches with
multiple strategies such as longitudinal studies of
students acting as their own controls over time is the
present attempt to respond to current concerns. In
addition, intervention students in combination with
matched groups at multiple intervals is a further
attempt to provide additional confidence in results.
Want a copy of the paper or
PowerPoint??
 [email protected]