Transcript Washington State LASER Study of Highly Successful Schools

Washington State LASER
Highlights From Recent
Evaluation Studies
Dave Weaver
RMC Research Corporation
111 SW Columbia, Suite 1200
Portland, Oregon 97201
December 2007
RMC Research Corp.
1
Agenda For This Segment
► Impact
on Student Achievement Studies
 Student-Level analysis of 2006 Science WASL
results
 School-Level analysis of 2007 Science WASL
results
 West Valley School District
► Implementation
Study
 Study of Highly Successful Schools
► Evaluation
December 2007
activities for 2007-2008
RMC Research Corp.
2
Student-Level Analysis
of 2006 Science WASL
Results
Spring & Summer 2007
December 2007
RMC Research Corp.
3
Student-Level Analysis of Grade 5
WASL Data for 2006
► Factors
considered
 Student demographics (i.e., socioeconomics,
race/ethnicity)
 Grade 5 science WASL results for 2006
 Grade 4 math and Reading WASL results for 2005
 The number of LASER PD hours the student’s Grade
5 teacher received each year and prior to the
administration of the 2006 WASL
 The number of LASER PD hours the student’s Grade
4 teacher received each year and prior to the
administration of the 2005 WASL.
December 2007
RMC Research Corp.
4
2-Years of Teacher Influence
2006 StudentLevel Science Data
for Grade 5
Linked By Teacher
PD Hours of
Student’s Grade 5
Teacher in 2006
Linked By Teacher
PD Hours of
Student’s Grade 4
Teacher in 2005
Linked By Student
2005 StudentLevel Math Data
for Grade 4
December 2007
RMC Research Corp.
5
Grade 5 Regression Analysis Results
Variable
Beta
t
p
FRL
–0.097
–23.389
<0.001
Hispanic
–0.020
–3.467
0.001
Asian
0.033
6.811
<0.001
White
0.070
11.160
<0.001
2005 Math Scale Score
0.688
174.346
<0.001
06 & 05 Teacher PD
0.024
6.496 <0.001
Dependent Variable: 2006 Grade 5 Science Scale Score
December 2007
RMC Research Corp.
6
Factors Influencing 2006 Grade 5
Science WASL Results
32,228 Students
860 Schools
1325 Classes
R2 = .645
(FRL)
9.7%
White
7.0%
(Hispanic)
2.0%
Asian
3.3%
LASER PD
2.4%
Others
6.8%
Grade 4
Scale Score
68.8%
December 2007
RMC Research Corp.
7
Grade 8 Regression Analysis Results
Variable
FRL
Beta
t
p
–0.073
–9.559
<0.001
0.064
8.512
<0.001
2005 Math Scale Score
0.747
100.039
<0.001
06 & 05 Teacher PD
0.038
White
5.369 <0.001
Dependent Variable: 2006 Grade 8 Science Scale Score
December 2007
RMC Research Corp.
8
Factors Influencing 2006 Grade 8
Science WASL Results
7,432 Students
R2 = .634
(FRL)
7.3%
White
6.4%
LASER PD
3.8%
Others
7.8%
Grade 7
Math Scale
Score
74.7%
December 2007
RMC Research Corp.
9
Finding 1
► After
controlling for demographics and
entry skill levels of students, the results
reveal that the current year of LASER
professional development is a significant
positive predictor of scores on the science
WASL results.
► If all things remain equal, 25 hours of
professional development translates into a
1-point gain in the science scale score.
December 2007
RMC Research Corp.
10
Finding 2
► The
professional development provided
during the year that the teacher had the
student in class had the greatest impact on
student performance.
► The accumulated effect of previous years of
professional development diminished
predictive value.
December 2007
RMC Research Corp.
11
Finding 3
► The
current year of professional
development of the 4th and 5th grade
teachers is a significant positive predictor of
student subscale scores for:
 Inquiry in Science (.039)
 Open-ended responses (.029)
 Applications of Science (.020)
December 2007
RMC Research Corp.
12
Finding 4
► Factors
beyond the control of LASER (entry
skills, demographics, poverty) account for
90% of knowable variance.
► Changes in any of these other factors can
easily overshadow the positive effect of
LASER professional development.
December 2007
RMC Research Corp.
13
School-Level Analysis
of 2007 Science WASL
Results
Fall 2007
December 2007
RMC Research Corp.
14
What We Did
►
Data Sources
 School Level WASL Data from 2007
 LASER Professional Development Database
 School Demographic Data for 2006-07
►
Regression Analysis
 Independent Variable:
► Total
PD hours of core teachers in a school for 2 years prior to 2007
WASL per FTE
 Dependent Variable:
► Percent
of students in the school that met the Grade 5 science
standard on the 2007 WASL
 Control Variables:
► Percent
of students who qualify for free or reduced price lunch
► Percent of students in the school that met the Grade 4 math standard
on the 2006 WASL
► Students per classroom teacher
December 2007
RMC Research Corp.
15
Grade 5 Regression Analysis Results
Variable
2006 Grade 4 Math
Beta
t
p
0.481
11.294
<0.001
FRL
–0.400
–9.425
<0.001
Class size
-0.088
-2.710
0.007
2-Year PD Per FTE
0.098
3.049
0.002
R2 = .617
December 2007
RMC Research Corp.
16
Findings
► The
number of hours of LASER professional
development per FTE over a 2 year period
prior to the WASL is a significant predictor
of student achievement on the Grade 5
science WASL above and beyond what can
be explained by the number of students
who qualify for free or reduced price lunch,
class size, or student skills.
► This finding was not evident at Grade 8.
December 2007
RMC Research Corp.
17
Percent of Grade 5 Students
Who Met Science Standard
2-Year PD Hours Per FTE vs.
Student Achievement
50
45
40
35
30
25
20
0
5
10
15
20
25
30
35
40
2-Year PD Per FTE
December 2007
RMC Research Corp.
18
Quartile Grouping of Schools
PD Per FTE Rank
Schools
Adjusted Mean
3.75 or less
85
30.56%
3.75 to 7.69
98
32.73%
7.69 to 13.42
99
34.50%
More than 13.42
103
37.01%
December 2007
RMC Research Corp.
19
Percent of Students Who Met
Science Standard
Total School 2-Year PD Hours Per FTE Quartile
Ranking vs. Percent of Students Who Met The
Grade 5 Science WASL in 2007
40%
Note: Means adjusted for Percent of students on
free or reduced price lunch, class size, and previous
year mathematics achievement.
35%
37.01%
34.50%
32.73%
30.56%
30%
25%
20%
3.75 or Less
3.75 to 7.69
7.69 to 13.42
More Than 13.42
PD Hours Per FTE
December 2007
RMC Research Corp.
20
Distribution of Study Schools By
LASER Alliance
Southwest
14%
Mt. to Harbor
3%
North Central
4%
North Sound
16%
Southeast
10%
Northeast
4%
Northwest
12%
South Sound
25%
South Central
12%
December 2007
RMC Research Corp.
21
Analysis of West
Valley School District
Data
Fall 2005
December 2007
RMC Research Corp.
22
West Valley School District study
► Peter
Finch, Assistant Superintendent
► Pilot
test involving phased implementation
of inquiry-based instructional materials
► Noticed
an important relationship between
the number of modules students
experienced and Grade 5 science WASL
scores.
December 2007
RMC Research Corp.
23
Science WASL – WVSD Grade 5
% students meeting standard
Students receiving LASER Science instruction
100
90
80
% Met Standard
70
60
State Average
50
48
45
43
40
2004
2005
30
27
30
19
20
10
0
None
1
2
3
Modules
December 2007
RMC Research Corp.
24
Regression Analysis of 2004 Grade 5
Science WASL Results
Variable
Coefficient
Significance
White
0.18
<0.001
SPED
-0.14
0.009
FRL
N/A
Modules
0.15
December 2007
RMC Research Corp.
0.003
25
Regression Analysis of 2005 Grade 5
Science WASL Results
Variable
Coefficient
Significance
White
0.18
<0.001
SPED
-0.34
<0.001
FRL
-0.16
<0.001
Modules
0.18
<0.001
December 2007
RMC Research Corp.
26
Conclusion
The number of science modules that
students experience has a significant
positive impact on their science
WASL achievement above and
beyond what can be explained by
demographic factors.
(FRL, Special Education, & %White)
December 2007
RMC Research Corp.
27
Study of Highly
Successful Schools
Winter & Spring 2007
December 2007
RMC Research Corp.
28
Purpose of Study
► To
identify schools that have demonstrated
a consistent increases in the percentage of
students who met the science standard on
the Washington State Assessment of
Student Learning (WASL).
► Interview the staff of those schools to
determine to what they attribute their
success.
December 2007
RMC Research Corp.
29
Criteria for Selecting of Schools
► Focus
on Grade 5
 Very few schools could demonstrate consistent
improvement at Grade 8 and 10.
► At
least 10 Grade 5 students complete the science
WASL in 2004, 2005, and 2006.
► An increase of at least 6% or more students
meeting the standard each year.
► An increase of at least 17.5% or more students
meet the science standard across the 2 time
intervals.
► Started with at least 20% of the students meeting
the science standard in 2004.
December 2007
RMC Research Corp.
30
Description of Selected Schools
► 40
schools selected for interview
► 18 (45%) were in districts served by a LASER
Alliance.
► 31 (78%) were in districts that participated in at
least 1 LASER Strategic Planning Institute
► 10 (25%) are in districts that participate in one of
the state MSP projects
► 4 (10%) are participating in the NCOSP
► 1 school did not participate in the interviews
December 2007
RMC Research Corp.
31
Interviews
► During
December 2006 and January 2007.
► Conducted By:
 RMC Research staff
 Graduate students from the science education
department of Portland State University
► Using
structure interview protocol
 Developed with input from LASER project leadership
 Training provided
► With
person most knowledgeable about success
 Initial contact with Principal and sometimes other school
staff
December 2007
RMC Research Corp.
32
Two Tiered Approach
► Open
ended initial question
 Summarized increases in WASL scores.
 What is your school doing to improve science teaching
and learning? To what do you attribute this success?
► Followed
by questions about the contribution of
specific aspects such as




Professional development
Instructional materials
Assessments
LASER services
December 2007
RMC Research Corp.
33
Finding 1– Instructional Materials
► All
schools used inquiry-based instructional
materials – Mostly FOSS
► 49% (19) of the schools indicated that recent
changes to the science instructional materials
attributed to their success
► Some customized or locally developed kits
► Heavily supplemented
► Non-LASER Alliance schools use a wider variety of
supplementary materials and approaches.
December 2007
RMC Research Corp.
34
Finding 2 – WASL Preparation
►Initially,
54% (21) of the schools
indicated that they engage students in
some form of preparation for the
WASL.
►Upon questioning, 95% (37) indicated
that they engage students in some
form of preparation for the WASL.
December 2007
RMC Research Corp.
35
Finding 2 – WASL Preparation
►Most
use WASL release items
 Prepare students before WASL
administration
 Develop WASL-like practice assessment
during the year
►Explicit
emphasis on
 WASL “format”
 WASL vocabulary
December 2007
RMC Research Corp.
36
Finding 3 – Professional Development
► 82%
(32) of the schools indicated that
professional development contributed to improving
student WASL scores
► PD in Non-LASER Alliance schools dominated by kit
and WASL training
► PD in LASER Alliance schools is more varies







Journaling/Notebooks
Leadership
Science Content
Inquiry
Coaching, Mentoring, & Consultant
EALRs and GLEs
More Organized School-Based PD
December 2007
RMC Research Corp.
37
Finding 4 – Science Advocate
► 77%
(30) of schools interviewed identified 1 or 2
people who were advocates for science education.
► Passionate
► Roles





about science – Made things happen
included:
Science specialist (16)
Teacher leaders (9)
Strong teacher or team (6)
Community member (3)
Administrator (2)
December 2007
RMC Research Corp.
38
Finding 5 – EALRs & GLE
►51%
(20) of the schools attributed the
success of their students to increased
emphasis on and alignment of
curriculum to the EALR’s and GLEs.
 11 are Non-LASER Alliance schools
 9 are LASER Alliance schools
December 2007
RMC Research Corp.
39
Finding 6 – Instructional Time
►38%
(15) of the schools attributed
their success at least partially to
increased science instructional time
 8 are Non-LASER Alliance schools
 7 are LASER Alliance schools
December 2007
RMC Research Corp.
40
Recommendations From Schools
►Interviewees
were asked:
 What 2 recommendations would you give
to the staff of another school working to
improve science teaching and learning
that would improve student performance
on the science WASL?
►Responses
December 2007
varied considerably
RMC Research Corp.
41
Top 6 Recommendations
► Use
released items & PCAs to develop skills
needed to be successful on the WASL (18).
► Provide adequate professional development
(13) on materials and WASL.
► Increase science instruction time (10).
► Engage students in hands-on, inquiry-based
science activities (9).
► Provide teacher collaboration time (7).
► Align with EALRs and GLEs (6).
December 2007
RMC Research Corp.
42
Conclusions
► LASER
actively promotes most of the key
findings from this study






Inquiry-based instructional materials
Preparing students for the science WASL
Professional development
Science advocacy (leadership)
Alignment of curriculum to EALR’s & GLEs
Increased science instructional time
December 2007
RMC Research Corp.
43
Evaluation Plans for
2007-08
December 2007
RMC Research Corp.
44
Sentinel Site Visits
► Identify
of approximately 30 schools in
2007–08 and 30 in 2008–09
► Schools With Significant LASER
Participation
► Two Groups of 15 schools each
 Demonstrated significant positive gains
 Demonstrated little, no, or negative gains
► Each
December 2007
site visit: 1 ½ to 2 days
RMC Research Corp.
45
Site Visit Data Collection
► Interviews
with principals and/or science teacher
leader
► Interviews with teachers
► Observations of at least 3 lessons where inquiry–
based instructional materials are used
 Based on STAR Protocol
► Rubrics
for rating level of implementation of
inquiry-based science
► Analyze results for significant relationships
between level of implementation to student
achievement
December 2007
RMC Research Corp.
46
Assessment of Student Content
Knowledge
► Obtain
permission to use Bellevue School
District assessments for Grades 6 & 7.
► Identify teacher participants
 25 first time teachers & 25 veteran teachers
► Data
collection for each student
 Pre and post student scores
 Student demographics
 Special program participation
► SPED,
► Gift
December 2007
FRL, ELL, Migrant, etc.
certificate compensation to teachers
RMC Research Corp.
47
Questions !
Dave Weaver
RMC Research Corporation
111 SW Columbia, Suite 1200
Portland, Oregon 97201
(800) 788-1887
[email protected]
December 2007
RMC Research Corp.
48