Transcript Daily PPT - Welcome to RMC Portland

Lessons Learned from Evaluating
Science Education Projects
Dave Weaver
RMC Research Corporation
111 SW Columbia, Suite 1200
Portland, Oregon 97201
[email protected]
(800) 788–1887
Presentation Contents
• Lessons Learned from LASER Sentinel
Site Visits
• 3 Common Pitfalls to Improvement
– How LASER and Washington Science
Leadership is Responding
• Implications for Your Strategic Planning
2
Analysis of Science WASL Data
• Evaluation Question
– To what extent did teacher professional
development on inquiry-based science instruction
contribute to improved student achievement on
the Grade 5 and 8 Washington Assessment of
Student Learning of science (science WASL)?
3
Finding 1
• The number of professional development hours in
which a student’s science teacher participated was a
small but significant predictor of student performance
on the science WASL above and beyond what could be
explained by socioeconomics (FRL) and the student’s
skill level (previous math WASL).
• This finding is consistent with earlier studies.
4
Percent of Grade 5 Students Who
Met/Exceeded Science Standard
PD Has Modest Impact on Achievement
50%
48%
46%
44%
42%
40%
38%
36%
34%
32%
30%
Less Than or Equal to
6.4 Hours
43.4%
40.6%
41.3%
Greater Than 6.4 and
Less Than or Equal to
15.75 hours
Greater Than 15.75
hours
Means adjusted by
previous year math
scores, and FRL.
Adjusted Means
3-Year PD Per FTE
5
Lessons Learned from
LASER Sentinel Site Visits
Results from the 2007-2009
LASER Biennium
Sentinel Site Selection
• Schools with significant LASER participation
during the 3 years prior to the site visit
• Schools visited
– 34 schools during 2007-08 school year
– 30 schools during 2008-09 school year
• Used standard protocol and rubrics
• Defined 2 groups of schools based on science
WASL change for 2 years prior to site visit
– Demonstrated Significant Positive Gains
– Demonstrated Little, No, or Negative Gains
7
Sentinel Site Visits
• RMC Research staff or consultants
• Each site visit: 1 ½ to 2 days
– Principal interview
– At least 5 teacher survey & interviews
– At least 3 classroom observations
• Data collection instruments
– http://www.rmccorp.com/LASERSiteVisits/
• Conducted web-based training session
• Results reported as rubric scores
8
Finding 2
• There were a number of traits that
site visitors consistently rated high,
many of which were a direct outcome
of Washington State LASER.
9
What Was Going Well At The School Level
•
Materials Support System
–
•
Condition of Modules
–
•
Strong evidence indicated that the district administrators were very supportive of inquirybased science instruction.
Sequence
–
•
Strong evidence indicated that the school administrators were very supportive of inquirybased science instruction.
District Support
–
•
The school was implementing 3 or more modules per grade level as the core science
curriculum materials. Ninety-four percent of the teachers used the modules as the core
science curriculum.
Administrative Support
–
•
Teachers always received modules that were complete and ready for classroom use.
Inquiry-Based Materials
–
•
The system for maintaining, storing, and refurbishing the instructional modules was
effective and well organized.
All science teachers used the modules according to a sequence clearly prescribed by the
district.
Critical Mass
–
Most (80% or more) teachers in the school had attended the initial use training for all of
10
the modules they used.
School Level Areas Receiving Low Scores
•
Summative District Assessments
– Very few school had districtwide or schoolwide summative assessments in
science that were administered to students annually
•
Formative Assessments
– A few teachers (25% or less) had adopted a standard formative assessment
strategy for science.
•
Instructional Time
– Science instructional time varied considerably amount teachers at the
elementary level and few schools required a minimum amount of
instructional time for science.
•
Professional Development Time for Teachers
– Teachers rarely had scheduled time during normal work hours to participate
in school-based professional development in science.
•
Partnership With Business, Informal Science, or Higher Education
– A few teachers had a tentative partnership with a business, an informal
science organization, or an IHE around science education.
11
Classroom Observation: High Scoring
•
Alignment of Lesson Activities
– Lesson activities addressed the stated learning objectives but there was
some question about how the lesson activities would lead to a deeper
student understanding of the learning objectives.
•
Motivation
– The lesson provided mostly extrinsic and some intrinsic motivation. The
intrinsic motivation was truncated by the lesson structure and was relatively
short lived.
•
Understanding of Purpose
– Throughout the lesson, many students understood why they were doing
each activity but the purpose of activities could have been more explicit.
•
Classroom Discourse
– For the most part, students and teachers support and encourage respectful
and constructive discourse around important science concepts, however,
only some students feel comfortable asking questions, backing up their own
claims, and/or critiquing claims made by others.
12
Classroom Observation: Low Scoring
• Lesson Closure
– By the end of the lesson, the teacher provided a brief review,
but students did not have an opportunity to fully make sense
out of how the lesson related to science concepts.
• Application of Science
– A few students applied something they learned in the lesson
to a new context.
• Reflection and Meta-cognition
– By the end of the lesson, students had some opportunity to
reflect on their thinking but students were not asked to
identify ways in which their thinking about the science
concepts had changed.
13
Analysis of Gains
• Divided the schools into 2 groups
– Based on:
• Percent of students who met the science standard
• Gains calculated over 2 years prior to the site visit
– Group definition
• Schools that demonstrated an increase in student achievement
• Schools that had no change or decreasing student
achievement
• The 2 groups of schools had significant demographic
differences
• Percent of students who qualified for free or reduced price
lunch
• Percent of Asian students
14
Finding 3
• Although schools that demonstrated
increasing student science
achievement were significantly
different demographically from those
that did not, there were also
significant differences in the
instructional practices of the teachers
observed by the site visitors.
15
Differences Between Gain Groups
• Science classes in schools that demonstrated an
increase in student achievement were more likely to:
–
–
–
–
Have clear lesson objectives
Involve activities that clearly address the learning objective
Have students who understand the purpose of the lesson
Have students that are more intellectually engaged in the
science content
– Have students applying science content to new contexts
– Have students engaged in science discourse
– Motivate students intrinsically
16
Analysis by Achievement Ranking
• Divided the schools into 2 groups
– Based on percent of students who met the science standard
the year of the site visit
– Group definition
• Schools at or above the state average
• Schools below the state average
• Methods
– Regression analysis to determine which variables were
significant predictors of the achievement ranking
– Controlling for
• Student achievement the previous year
• Percent of students who qualify for free or reduced price lunch
17
Finding 4
• Although schools that demonstrated
above average student science
achievement were significantly
different demographically from those
that were below average, there were
also significant differences in the
characteristics of the schools.
18
Characteristics of Above Average
Performing Schools
•
Instructional Time Allocated
– The elementary school has a designated amount of instructional time
allocated for science.
•
Professional Development Time for Teachers
– Teachers occasionally had scheduled time during normal work hours to
participate in school-based professional development in science.
•
District Support
– Evidence indicated that the district administrators were somewhat
supportive of inquiry-based science instruction.
•
Integration of Literacy
– Many teachers integrated science and literacy through the use of
supplementary reading materials and science notebooks.
•
Parent and Community Support
– Evidence indicated that the parents/community were somewhat supportive
of inquiry-based science instruction.
19
Principal Survey
• Online survey that closely paralleled data collected
during sentinel site visits
• Selected schools with 15 hours or more of science
professional development per teacher over a 5-year
period prior to March 31, 2009
• 319 schools invited
• Survey administered between May 22 and July 31,
2009
• 62 principals completed the survey (19.4% return rate)
20
Principal Survey Analysis
• Regression analysis to determine which
survey items were significant predictors
of student achievement on the 2009
science WASL
• Controlling for the percent of students
who qualified for free or reduced price
lunch (FRL)
21
Finding 5
• There were several items on the
principal survey that were
significant predictors of student
performance on the science
WASL above and beyond what
could be attributed to FRL.
22
Predictors of Student Performance
• Schools that made an organized effort to identify
instructional materials to fill the gaps
• Schools where the principals observe student using
evidence to engage in discourse about science.
• Schools that provide time during the normal work day
for professional development and how often teachers
participate in school-based science professional
development.
• Schools that support professional learning
communities that focus on improving science
teaching and learning.
23
Regression Analysis Results
Grade
Tested
Adjuste
d R2
Change
in R2
p
Beta
Has your school or district made an organized effort to
identify instructional materials to fill the gaps?
5
.459
.114
<.001 *
.352
Principal observation of classes: Students had
opportunities to make claims, and/or use evidence to
back up their claims or critique claims made by others.
The lesson reinforced the notion that science is a process
by which knowledge is constructed.
5
.563
.122
<.001 *
.350
Is time scheduled during normal work hours for teachers
in this school to participate in organized, school-based
professional development specifically for science?
5
.338
.002
.003 *
.048
How often do teachers participate in school-based
professional development specifically for science?
5
.213
.017
<.001 *
.153
Has any of the PLC activities focused on improving
science teaching and learning?
5
.476
.031
<.001 *
.211
Approximately what percentage of the PLC is devoted to
science teaching and learning?
5
.434
.002
.013 *
.047
Survey Item
24
Finding 6
• There are several consistent themes
among the various findings that
provide the basis for recommendations
and conclusions that lead to a shift in
Washington State LASER.
25
Significant School-Level Factors
•
•
•
•
•
•
PD time for teachers during the school day
Allocated science instructional time
Professional learning communities
Filling curriculum gaps
Integration of literacy
District, parent, and community support
26
Significant Instructional Factors
•
•
•
•
•
Science discourse using evidence
Purposeful instruction
Intellectual engagement
Intrinsic motivation
Application of science skills
27
Conclusion
• The infrastructure to support the use of a core
curriculum of inquiry-based science instructional
modules is in place and is functioning
adequately in the schools visited.
• Although these conditions are necessary for
the implementation of inquiry-based science
instruction, they are not sufficient to raise
student achievement as measured by the
science WASL.
28
Recommendation 1
• Ensure that the professional development on
research-based instructional practices is
consistent and explicit across all of the
LASER Alliances
– Help teachers understand the elements of effective
science instruction and use the modules as a
means of carrying out the element with their
students.
29
Recommendation 2
• Increase support for school-based
professional development that helps
teachers:
– Assume accountability for student learning
that results from the use of the modules,
and
– Collaboratively implement the elements of
effective science instruction.
– Ample structure and leadership for success
30
3 Common Pitfalls to
Improvement
Common Pitfalls
• Why haven’t past school reform
efforts been more successful?
• Three Common Pitfalls:
– Lack of Vision
– Lack of Process
– Atomized Culture
32
Common Pitfall #1
• Lack of Vision
– “In most instances, principals, lead teachers, and
system-level administrators are trying to improve
the performance of their schools without knowing
what the actual practice would have to look like to
get the results they want at the classroom level.”
(City, 2009, p 32)
– There is often a “lack of an agreed-upon definition
of what high-quality instruction looks like.”
33
Education Has Not Been Scientific
• Education has been criticized for not
being “scientific.”
• We do know a lot about how students
learn various subjects.
• The problem is that this knowledge is
not consistently applied in daily
instructional practices.
34
Addressing #1—Lack of Vision
• “Develops, with colleagues who have to work
together on school improvement, a shared
understanding of what they mean by effective
instruction.”
(City, 2009, p 10)
• Build vision of effective instruction upon
knowledge of how students learn the subject
– Most instructional practices fail to effectively apply
what research tells up about how students learn a
given subject.
35
Examining Research
Reading Research Converges
• Reading—50 years of research
• Effective reading instruction requires a
balanced blend of:
•
•
•
•
•
Phonemic awareness
Decoding
Vocabulary development
Reading fluency, including oral reading skills
Reading comprehension strategies
• Any single approach is inadequate
37
Math Research Converges
• National Math Panel Report
• Effective mathematics instruction
requires a balanced blend of:
• Computational fluency
• Concept development
• Problem solving
• Any single approach is inadequate
38
What About
Science?
39
Science Is Different From
Language Arts and Math
• Language Arts (& Reading) and Math
– Are skills created by people
– Involves learning established conventions
• Science
– Understanding how the world works
– How to create new knowledge
– Living in the world creates a working understanding
which may or may not be scientifically valid
40
Importance of Learning Theory
in Science
• Working knowledge is entrenched and
difficult to overcome
• Sometimes called “Private Universe”
• Key concepts must be internalized
– Sometimes called “Big Ideas” or “Enduring
Understandings”
• Requires greater attention to the learning
theory embodied in the instructional materials
41
Research On Effective Science
Instruction is Also Converging
• Considerable evidence from research shows
that instruction is most effective when it
elicits students’ initial ideas, provides them
with opportunities to confront those ideas,
helps them formulate new ideas based on
evidence, and encourages students to reflect
upon how their ideas have evolved.
(Banilower, 2009)
42
Untrenching Their Private Universe
• Without these opportunities, students
“may fail to grasp the new concepts and
information that are taught, or they may
learn them for purposes of a test but
revert to their preconceptions outside
the classroom”
(National Research Council, 2003, p. 14)
43
Therefore:
• Effective science instruction must be
guided by what research tells us about
how students learn science
• It is the duty of all science teacher to
enact sound learning theory in science
instruction
LASER Theory of Action
• If teachers use research-based instructional practices,
materials, and assessments so that students:
– Draw upon a deep foundation of usable knowledge within the
context of a conceptual framework to build scientific understanding
– Are intellectually engaged and motivated
– Reveal preconceptions and their initial reasoning
– Use evidence to generate explanations
– Communicate and critique their scientific ideas and the ideas of
others
– Have full access to activities and sense-making discussions to
develop scientific understandings
– Reflect on how personal understanding has changed over time
and recognize cognitive processes that lead to changes
45
Then:
• The percent of students who meet or exceed the
science standards on the state and other science
assessments will increase,
• Student participation and success in rigorous science
courses K–12 will increase,
• The number of students who seek further studies in
STEM content and STEM careers will increase, and
• The number of students who choose to do sciencebased activities out of school time will increase.
46
In Other Words
Student science
achievement will increase if
we consistently apply what
we know about how
students learn science!
47
Common Pitfall #2
• Lack of a Process
– Teachers attend professional
development but then it is up to
individuals to put the new knowledge
and skills into practice
– “[Schools] don’t have a process for
translating that knowledge
systematically into practice.”
48
Addressing #2—Lack of Process
• Establish a school-based collaborative process
that engages teachers in:
– Regular structured observations of students for
evidence that the vision of how students learn the
subject is being enacted
– Engage teachers in a process to make meaning from
the observation experience to continuously improve
instructional practices
• “Puts educators in a position of having to
actively construct their own knowledge of
effective instructional practice”
49
(City, 2009, p 10)
Common Pitfall #3
• Atomized Culture
– “Most people in schools work in siloed cultures
characterized by independence and autonomy.”
(City, 2009, p 62)
– Too often schools “are organizations that support
the private practice of teachers” and the atomized
culture.
– Atomized culture is one of the most significant
barriers to school improvement.
50
Addressing #3—Atomized Culture
• Deprivatize Practice
– “It is clear that closed classroom doors will not help
us educate all students to high levels.”
– “Everyone is obligated to be knowledgeable about
the common task of instructional improvement and
everyone’s practice should be subject to scrutiny,
critique, and improvement.”
– “We can do more together than we can individually
to improve learning and teaching.”
(City, 2009, p 3)
51
Addressing #3—Atomized Culture
• Importance of Observation
– “The only way to find out what
students are actually doing is to
observe what they are doing.”
– “What predicts performance is what
students are actually doing.”
(City, 2009, p 30)
52
Implications for Planning
• Things to Remember
– The infrastructure and initial use training to
support the use of a core curriculum of
inquiry-based science instructional
modules is necessary but not sufficient
to raise student achievement in science
– Fill curriculum gaps
53
Implications for Planning
• Embrace the LASER vision of effective
science learning experiences for students
embodied in the LASER theory of action
– Ensure all professional development focuses on
helping teachers enact that vision
– Implementing the effective learning experiences
for students is the purpose of professional
development
54
Implications for Planning
• Establish an internal process for continuously
improving instruction that:
–
–
–
–
–
–
–
Provides time within the school day for participation
Has sufficient structure
Has creditably leadership
Provides access to content expertise
Deprivitizes practice
Involves observation
Strives for horizontal and vertical articulation
55
Questions ???
Dave Weaver
RMC Research Corporation
111 SW Columbia, Suite 1200
Portland, Oregon 97201
[email protected]
(800) 788–1887
Presentation is at:
http://www.rmccorp.com/LASER/
56
“Must Read” List
• City, E., Elmore, R., Fiarman, S., & Tietel, L.
(2009). Instructional rounds in education.
Cambridge, MA: Harvard Education Press.
• Banilower, E., Cohen, K., Pasley, J., & Weiss, I.
(2008). Effective science instruction: What does
research tell us? Portsmouth, NH: RMC
Research Corporation, Center on Instruction.
• National Research Council. (2003). How people
learn: Brain, mind, experience, and school. J. D.
Bransford, A. L. Brown, & R. R. Cocking (Eds.).
Washington, DC: National Academy Press.
57