Washington State LASER 2008 Evaluation Report

Dave Weaver Kelly Jarvis

RMC Research Corporation 111 SW Columbia, Suite 1200 Portland, Oregon 97201 [email protected]

(800) 788 –1887

Presentation Contents

• Analysis of 2007 Science WASL Results – School-Level Analysis – Student-Level Analysis – Longitudinal Analysis • Module-Level Assessment Results • Sentinel Site Visits Preliminary Results • Recommendation • Upcoming Activities 2

Analysis of 2007 Science WASL Results

• Evaluation Question –

To what extent did teacher professional development on inquiry-based science instruction contribute to improved student achievement on the Grade 5 and 8 Washington Assessment of Student Learning of science (science WASL)?

3

School-Level Analysis

• Data Sources – School-Level Science WASL Data for 2007 – School-Level Mathematics WASL Data for 2006 – School-Level Demographic Data – School-Level Professional Development (PD) Index • Total PD Hours of Teaching Staff for 2-Years Per FTE • Methods: Linear Regression Analysis – Dependent Variable: • Percent of Students Who Met Science Standard on 2007 WASL – Independent Variable: • School-Level 2-Year PD Index – Control Variables: • Percent of Students Eligible for Free or Reduced Price Lunch (FRL) • Students Per Classroom Teacher 4

Student-Level Analysis

• •

Data Sources

– 2007 Student-Level Science WASL Results for Grade 5 & 8 with Student Demographics – 2006 Student-Level Mathematics WASL Results for Grades 4 & 7 – Teacher-Level 3-Year PD Index • Total PD Hours Teacher for 3-Years Prior to WASL

Methods:

Linear Regression Analysis – Dependent Variable • 2007 Science WASL Scale Score – Independent Variable • Teacher-Level 3-Year PD Index – Control Variables • FRL • Previous Grade Mathematics WASL Scale Score 5

Student-Level Data Map

6

Longitudinal Analysis

• Data Sources – 2007 Student-Level Science WASL Results for Grade 8 with Student Demographics – 2004 Student-Level Science WASL Results for Grades 5 – School-Level 3-Year PD Index • Total PD Hours Teacher for 3-Years Prior to WASL • Methods: Linear Regression Analysis – Dependent Variable • 2007 Science WASL Scale Score – Independent Variable • School-Level 3-Year PD Index – Control Variables • FRL • Previous Grade Mathematics WASL Scale Score 7

Longitudinal Data Map

8

Finding 1

•

The number of professional development hours in which a student’s science teacher participated was a small but significant predictor of student performance on the science WASL above and beyond what could be explained by socioeconomics (FRL) and the student’s skill level (previous math WASL).

•

This finding is consistent with earlier studies.

9

School-Level Analysis Results

40% 38% 36% 34% 32% 30% 28% 26% 24% 22% 20% 30.1% 33.8% 34.6% 37.0% Less Than or Equal to 3 Hours Greater Than 3 and Less Than or Equal to 7.5 hours Greater Than 7.5 and Less Than or Equal to 13.4 hours Greater Than 13.4

hours Adjusted Means 2-Year PD Per FTE

Means adjusted by previous year math scores, class size, and FRL.

10

Longitudinal Analysis Results

400 399 398 397 396 395 394 393 392 391 390 394.8

394.3

395.5

397.1

Greater Than 0 and Less Than or Equal To 1.25

Greater Than 1.25 and Less Than 3 Greater Than or Equal to 3 and Less than 6.3

Greater than or Equal to 6.3

Adjusted Means School-Level PD Index Category (Secondary)

Means adjusted by Grade 5 Science Scale Score, FRL, Non-Minority, Special Education, and Gifted.

11

Finding 2

•

Grade 5 teachers must have participated in at least 18 hours of professional development before an impact on student achievement was measurable, and the impact of the professional development on student achievement increased as the number of professional development hours increased.

12

Grade 5 Multiple Regression Analysis Across Teacher-Level PD Index Categories

PD Index Interval

More than 0 but less than 9 hours At least 9 hours but less than 18 hours At least 18 hours but less than 36 hours At least 36 hours but less than 54 hours 54 hours or more

N

5,633 5863 7589 2740 1366

Beta

-.013

t

-1.410

.011

.025

.034

.093

1.279

3.176

2.658

5.009

p

.158

Adj. R 2

.542

SE

14.682

.201

.001* .008* <.001*

.584

.538

.547

.534

14.105

15.318

14.710

15.010

NOTE

: Dependent Variable is the Grade 5 science WASL scale score. Control variables are the previous year math scale score and FRL Results are weighted by the number of students assessed.

* denotes significance at the .05 level.

Finding 3

• •

Grade 5 teachers who had participated in science professional development served:

–

A larger proportion of students eligible for free or reduced-price lunch;

–

A greater proportion of Hispanic and Latino students; and

–

A smaller proportion of White students.

These differences may account for the differences in student achievement.

14

Differences in Demographics and Achievement of PD Participants vs. Non-Participants Variable

Total Number of Students

Student Achievement

Mean 2007 Science WASL Score Mean 2006 Math WASL Score

Ethnicity

Hispanic/Latino White (Not Hispanic Origin)

Special Program

Free or Reduced Price Lunch

No PD

41,400 391.17

409.26

12.3% 69.0% 37.1%

Any PD

30,006 389.53

406.71

17.6% 63.5% 43.1%

Module-Level Assessment of Student Content Knowledge

• Evaluation Questions – What science content knowledge gains do students make as a result of their use of the inquiry-based instructional modules?

– What is the relationship between the science PD of the teacher and the gains in student content knowledge?

16

Module-Level Assessment of Student Content Knowledge

• Assessment – Obtained Permission to use Horizon Research Assessment – Selected Horizon Research Instruments • Human Body from FOSS • Levers and Pulleys from FOSS • Rocks and Minerals from STC • Electric Circuits from STC 17

Module-Level Assessment of Student Content Knowledge

• Administration Plan – Recruit teachers with a range of PD levels – Data collection for each student • Pre and post student scores • Student demographics – Gift certificate compensation to teachers – Data collected spring 2008 • 53 classes completed both pre and post assessments • 1087 students 18

56 Teachers Recruited

Kit Electric Circuits Human Body Rocks and Minerals PD Level

High (>54 hours) Medium (>18 but <54 hours) Low (<18 hours) None High (>54 hours) Medium (>18 but <54 hours) Low (<18 hours) None High (>54 hours) Medium (>18 but <54 hours) Low (<18 hours) None

Total: Total: Total: Teachers

12 5 5 4

26

3 4 3 7

17

2 5 5 1

13

Finding 4 •

Students demonstrated significant improvement between the preassessment and the postassessment on all 3 modules: on average, student scale scores increased more than 19 points

.

20

Module-Level Student Assessment Results

Pre Post 100 90 80 70 60 50 40 30 20 10 0 42.7

62.1

42.5

62.4

37.4

56.5

Electric Circuits Human Body Rocks and Minerals

21

• •

Finding 5

The relationship between student performance and teacher professional development was inconclusive.

–

The performance of the students whose teachers participated in fewer than 18 hours of professional development was significantly lower than the performance of students whose teachers participated in 18 or more hours professional development.

–

Students whose teachers did not participate in professional development outperformed students whose teachers participated in some professional development.

Additional data will be collected during the 2008 –2009 school year.

22

Student Assessment Gains as a Function of Professional Development

Sentinel Site Selection

• Schools With Significant LASER Participation • Identified 36 Schools – 13 Schools Refused Participation – Added 11 New Schools – Visited 34 Schools • Defined 2 Groups of Schools Based on Science WASL Change Between 2006 and 2007 – Demonstrated Significant Positive Gains – Demonstrated Little, No, or Negative Gains 24

Sentinel Site Visits

• 10 Site Visitors • Each site visit: 1 ½ to 2 days – Principal Interview – At Least 5 Teacher Survey & Interviews – At Least 3 Classroom Observations • Data Collection Instruments – http://www.rmccorp.com/LASERSiteVisits/ • Conducted 3 web-based training session • Round 1: Spring 2008 • Results Reported As Rubric Scores 25

Finding 6

•

The site visitors gave the vast majority of the schools visited very high ratings in several areas indicating that the schools served by the LASER Alliances had:

–

Successfully established a core sequence of inquiry based instructional modules across all of the grade levels, and

–

Had materials support system that delivered modules to teachers on time and ready to use.

26

Finding 7 •

There were several significant differences between schools whose students demonstrated increased performance on the science WASL compared to schools whose students did not.

27

Schools with Grade 5 Students

• Schools whose Grade 5 students’ science WASL scores increased had significantly greater: – Participation in professional learning communities – Time during the workday for school-based professional development – Evidence of positive parent and community support – Evidence of teachers integrating literacy with science 28

Schools with Grade 8 Students

• Schools whose Grade 8 students’ science WASL scores increased had : – Time during the work day for school-based professional development – Greater participation in professional development that involved examining student work – Lessons that more often engaged students in

discussion based on scientific evidence

– Lessons whose content was more significant, accurate and worthwhile 29

Schools with Grade 8 Students (cont.)

• Schools whose Grade 8 students’ science WASL scores increased had: – Lessons that more often built on

prior ideas and experiences

– Lessons whose

content

was more closely

connected

the lesson

activities

– Lessons that were effective at

fostering

student to

understanding

– Lessons that included mechanisms to monitor students’ understanding of science content – Lessons that helped students

make sense

of scientific concepts – Lessons that routinely provided opportunities for students to

reflect

on their new understanding 30

• •

Finding 8

The schools that demonstrated an increase in the percentage of students who met the science standards had teachers with significantly fewer years of education experience than the schools with no gains.

This difference was more pronounced in the schools that served Grade 8 students.

– Testing for demographic differences between the schools that improved their science WASL scores and schools that did not could not account for the findings. Factors examined include: • School size • Ethnic/racial composition of the student population • Program membership of the students • Teacher education and experience 31

Conclusion

• The infrastructure to support the use of a core curriculum of inquiry-based science instructional modules is in place and is functioning adequately in the schools visited.

• Although

these conditions are necessary

for the implementation of inquiry-based science instruction,

they are not sufficient

to raise student achievement as measured by the science WASL.

32

Recommendation 1

•

Ensure that the professional development on research-based instructional practices is consistent and explicit across all of the LASER Alliances

– Help teachers understand the

elements of effective science instruction

and use the

modules as a means

of carrying out the element with their students.

33

Elements of Effective Science Instruction

• Intrinsic Motivation – Relevant, interest to student, resolution of dissonance • Eliciting Student’s Prior Knowledge – Articulating what they already believe about a concept • Intellectual Engagement – Students do the thinking • Use of Evidence – Discourse: Justifications and explanations based on evidence • Sense-Making – Closure, meta-cognition, reflection

Banilower, E., Cohen, K., Pasley, J., & Weiss, I. (2008). Effective science instruction: What does research tell us? Portsmouth, NH: RMC Research Corporation, Center on Instruction.

34

Recommendation 2 •

Increase support for school-based professional development that helps teachers:

–

Assume accountability for student learning that results from the use of the modules, and

–

Collaboratively implement the elements of effective science instruction.

–

Ample structure and leadership for success

35

Evaluation Activities for 2008-09

• Sentinel Site Visits (30 to 35) • Student Content Assessments – Fall, Winter, & Spring – 7 Instructional Modules • Analysis of 2008 WASL Data • Online Surveys – School Principal Survey – Alliance Directors • PD Provider Online Surveys (3)

Rocks & Minerals Human Body Electric Circuits Motion & Design Levers & Pulleys Environments Variables

36

ASK-IT Schools Show What Science Engaged Schools Can Do

ASK IT schools had teachers who:

Used formative assessment to determine student understanding of key concepts Met in professional learning communities to inform their teaching based on student data Increased their science teaching time so most teachers completed each unit.

0 -0.5

-1 -1.5

-2 Pre

Environments Proficiencies 5th Grade Matched Pair 1

Post Treatment Control

Magnetism & Electricity Overall Proficiencies Human Body Overall Proficiencies

0.5

0 -0.5

-1 -1.5

-2 Treatment Control 0 -0.5

-1 -1.5

-2 -2.5

Treatment Control Pre Post Pre Post 37

Questions ???

Dave Weaver

RMC Research Corporation 111 SW Columbia, Suite 1200 Portland, Oregon 97201 [email protected]

(800) 788 –1887 38