Transcript Comparing the Effects of Two Versions of Professional Development on Science Curriculum Implementation and Scaling-Up Session 43.030 April 13, 2005 American Educational Research Association Annual Meeting Montreal,

Comparing the Effects of Two Versions
of Professional Development
on Science Curriculum
Implementation and Scaling-Up
Session 43.030
April 13, 2005
American Educational Research Association Annual Meeting
Montreal, Canada
Introduction and Overview
Paul R. Brandon
Curriculum Research & Development Group
University of Hawai‘i at Mānoa
Project Purpose
• A randomized study of the effects of
variations in professional development
(PD) on
– program implementation
– student achievement
– scale-up
• First phase (ending 2/28/06) to prepare for
five-year second phase (if funded)
3
FAST
• The project compares two versions of
teachers’ PD for Foundational Approaches
in Science Teaching (FAST).
– An award-winning middle-school inquirybased science program
– Shown to positively affect achievement and
other student outcomes
4
Phase-I Project Tasks
• Prepare the 5-day PD institute with
electronic resource and on-line course
• Prepare instruments for the second phase:
– Teacher questionnaire
– Teacher log
– Observation procedures
– Student assessments
5
Focus on Span of PD
• Traditionally, FAST PD is delivered in a 10day institute.
• We will compare it with an alternative 5day institute followed by an online course,
with an electronic multimedia resource.
6
Phase-II Hypothesis
The 5-day inquiry-based science training
institute, followed by an on-line university-credit
course with computer-based multimedia, will
have more favorable outcomes than a 10-day
institute without the follow-up university course.
7
Examining the effects of PD on:
• levels of classroom implementation
• intensiveness and extensiveness of longterm use of inquiry-based science (i.e.,
scale-up)
• levels of student achievement
8
Experimental Design
• Randomized cluster sample, with schools
(N = 80) as clusters
• One teacher per school
• Even if attrition is 25%, statistical power
will be .80.
9
Merit of the Study
• Addresses at least five deficits in the
literature:
– Few studies on effects of PD time span
– Few studies about using technology in PD
– Few randomized studies
– Few studies of the effects of PD on student
learning
– Few, if any studies, of inquiry science PD
10
Order of Presentations
• Gray, Nguyen, & Speitel: Description of PD
• Brandon: Log and questionnaire development
and questionnaire validation
• Taum: Observation guide development
• Ayala: Student assessment development
• Lawton: Comparison of early effects of the two
versions of PD
11
Introduction and Overview
Paul R. Brandon
Curriculum Research & Development Group
University of Hawai‘i at Mānoa
Developing and Implementing an
Alternative Version of FAST
Professional Development
Mary E. Gray
ThanhTruc T. Nguyen
Thomas W. Speitel
University of Hawai‛i at Mānoa
FAST
• Foundational
Approaches in Science
Teaching
• Inquiry-based, middle
school science
• Physical, biological,
and earth sciences
• Exemplary program
(USDOE, 2001)
14
FAST Professional Development
• All FAST teachers must
participate in PD
– Certification required to
purchase science
curricula
•
•
•
•
•
Content
Inquiry
Standards
Assessment
Classroom organization
and management
• Safety
15
The Challenge
To develop and implement an alternative version
of FAST I professional development
16
Considerations
• Current needs of science teachers
• Future possibilities (emerging
technologies)
• Unique challenges (the dynamic nature of
inquiry)
17
Review of Literature
Professional Development
Inquiry Science
On-Line Learning
18
Review of Literature
Professional Development
• Strong subject area content (Kennedy, 1999)
• Focus on higher-order teaching strategies
(Porter, Garet, Desimone, Yoon, & Birman, 2000)
– Collaborative, Same subject-grade-school,
Active learning, Consistent with teacher goals
19
Review of Literature
Professional Development
• Quantity linked to improvements (Radford, 1998;
Supovitz, Mayer, & Kahle, 2000; Fishman, Marx, Best, & Revital, 2003)
– science content knowledge, process skills, and
attitudes
• Quantity linked to standards based teaching
(Supovitz & Turner,2000)
• Problems with researching inquiry-based
teaching and student learning (Fishman et al., 2003)
20
Review of Literature
On-line Learning
• 3 million adult learners on-line (Waitts & Lewis,
2003)
• Convenience and increased flexibility
(Hülsmann,1999)
• Evidence of implementation of various
pedagogical approaches (Schlager & Schank,
1997)
21
Review of Literature
On-line Learning
• Quantifiable learning outcomes were not
significantly linked to technology adoption (Jones
and Paolucci,1998).
• Focus on motivation, skills and knowledge, selfdirected learning, interactive competence, and
technology skills (Kabilan, 2004)
• Increased efficacy and self-perception in
teachers (Huai, Braden, White, & Elliot, 2003)
22
FASTPro
23
How are the traditional PD and
FASTPro different?
• Two-weeks in
duration, face-to-face
• One-week in duration,
face-to-face
• Conduct 43
investigations
• Conduct 19
investigations
• Extensive modeling
and practicing time
• Electronic resource
• WebCT-based
learning community
24
Inquiry
Teaching and learning science through
inquiry is a new experience for many
teachers and requires a significant
change in attitude and behavior.
FASTPro – Addressing Inquiry
• Strategies (Loucks-Horsley, Hewson,
Love, & Stiles,1998)
– Examples
• Immersion in inquiry
into science
• Coaching and
mentoring
• Technology for
professional learning
26
FASTPro – Addressing Inquiry
• Change (Hord, Rutherford, Huling-Austin, & Hall,1987; Guskey,
2000)
– takes time and persistence
• awkwardness and frustration expected
– as teacher’s progress through a change process,
their needs for support and assistance change
• Optimal Mix (Guskey, 2000)
– A combination of teacher, program and change
agent that will help create a positive relationship for
PD to be effective
27
FASTPro
28
FASTStart
• One week face-to-face
institute
• Teachers conduct
nearly half of actual
student investigations
• Use inquiry
methodology including
modeling, discussion,
assessment, and
practice
29
30
FASTeR
• Observe interactions
of students and
teachers
• View and reflect upon
investigations not
experienced
• See detailed step-bystep procedure
suggestions
• Movies and
animations
• Web interface
– Quicktime and Flash
plugins
• DVD-ROM medium
31
Examples from FASTeR
32
Examples from FASTeR
• Slideshow 9, Density
and the Cartesian
Diver
• Slideshow 22,
Identifying Unknown
Substances
33
34
FASTForward
35
Formative Findings
• Suggest that teachers
were able to implement
successfully and share
teaching practice
• Added utility, focused and
meaningful to teachers’
own environment
36
Future considerations
• Infuse technological
aspects into the
FASTStart face-to-face
experience
• Address credit
equivalencies to enable
more robust activity
requirements
• Expand to include FAQ’s,
indexing, and rich
descriptions
• Expand to include a
Website as well as DVDROM
• Automate some computer
assisted instruction and
feedback mechanisms
37
Reflections…
38
Reflections…
• Teamwork and
flexibility
• Coordination of
equipment and
resources
39
Future Considerations
• Building capacity
• Application to other
CRDG quality
educational
programs
40
Mahalo.
41
Developing and Implementing an
Alternative Version of FAST
Professional Development
Mary E. Gray
ThanhTruc T. Nguyen
Thomas W. Speitel
University of Hawai‛i at Mānoa
Instrument Development for a Study
Comparing Two
Versions of Inquiry Science
Professional Development
Paul R. Brandon
Alice K. H. Taum
University of Hawai‘i at Mānoa
Identifying Constructs
• on teaching science with inquiry methods
– Reviewed FAST and other inquiry science documents
and worked closely with inquiry science experts.
• on the context within which inquiry-based
science is taught.
– Reviewed 55 books and key articles on curriculum
indicators and school effectiveness.
44
Developing the Teacher Log
• Purpose: To identify the extent to which
teachers implement the key features of
inquiry-based science.
• Reviewed the recent literature on logs.
• Kept the instrument short to avoid
overburdening teachers (21 items).
• Is completed immediately after finishing
each student science investigation.
45
Focus of the Teacher Log
• Instrument addresses topics such as:
– students’ questioning behaviors.
– the teacher’s use of questioning strategies.
– the teacher’s circulation about the classroom.
– teacher-led discussions about variations in
the data.
46
Developing the
Teacher Questionnaire
• Purposes:
– To obtain information about implementation
that is most appropriately collected once a
year.
– To collect information on implementation of
investigations not covered on logs.
47
Focus of the Questionnaire
• 150 items address topics such as:
– implementation of key features.
– the investigations taught during the entire
year.
– the adequacy of materials, equipment, etc.
– teacher demographics.
– teacher attitudes toward science.
– teacher participation in science activities
outside the classroom
48
Types of Validation Studies to Date
• Two types of questionnaire-data validation
studies conducted to date:
– analyses addressing the relationship between
the responses on the log and the
questionnaire (“concurrent” validity)
– an analysis examining the theoretical model
underlying our study (construct validity)
49
Relationship Between Log and
Questionnaire Responses
• Compared results on total scores for five
items that the two instruments had in
common
• Correlation = .54; effect size = .50
50
Examining the Theoretical Model
• Second set of analyses examined whether the
expected relationships were found among some
questionnaire variables.
• We regressed a variable measuring program
implementation on five independent variables
measuring:
–
–
–
–
–
teachers’ ongoing learning about science.
the resources available for teaching science.
the number of science courses taken.
the number of years have taught K–12 science
teachers’ attitudes toward teaching science
51
Validity Argument
• Teachers’ ongoing learning about science
(including learning that occurred in recent
PD) should predict implementation more
than
– science education background.
– years of experience teaching science.
– attitudes toward teaching science.
– classroom resources available.
52
Results of Second Analysis
•
•
•
•
Teacher learning about science outside the classroom:
beta = .19 (significant at the .05 level)
Teacher attitudes toward teaching science:
beta = .17 (significant)
•
•
•
•
•
Classroom and school resources for teaching science:
beta = .04 (not significant)
Number of university science courses taken:
beta = -.02 (not significant)
Number of years the teacher has taught K–12 science:
beta = -.02 (not significant)
53
Instrument Development for a Study
Comparing Two
Versions of Inquiry Science
Professional Development
Paul R. Brandon
Alice K. H. Taum
University of Hawai‘i at Mānoa
Coding Teachers in
Science Classrooms using the
Inquiry Science Observation Guide
Alice K. H. Taum
Curriculum Research & Development Group
University of Hawai‘i at Mānoa
Inquiry Science Observation Guide
• Introduction and Overview
– FAST, SCUP Project, Purpose
and Development of ISOG
• General coding guidelines
• Descriptions for each of the six
Activities
• Definitions of Activity details
• Code Sheet
• Recording Sheet
• Reconciling Recording Sheet
56
Development of the
Code Sheet
• 1.5 year process
• Collaborative efforts between researchers
at the University of Hawai‘i at Mānoa,
Stanford University and Sonoma State
University; FAST teachers; curriculum
developers, and coders
• 36+ revisions
57
Goal of the Code Sheet
• Identify strings of activities/teacher
behaviors using the details in columns A,
B, C and D.
– questioning strategies used
– level of engagement between teacher and
students
– creating a profile of teachers implementation
of FAST in the classroom through science
inquiry
58
Design of the Code Sheet
• 6 activities.
• 66 possible activity details to plug into the
6 activities.
• Multiple variations between an activity and
the activity details which describe an
activity string of teachers pedagogical
practices.
59
Inquiry Science Observation Code Sheet (ISOCS)
Activity
1-6
Activity
descriptors
Activity details
A
B
C
D
1. Teacher directs student (s)
1. Teacher directs student(s)
______A______
to
_____A_______ to
______B______
_____B_____
_______C______.
_____C_____.
1A1. individually
1A2. in a small group
1B1. record
1B2. discuss
1B3. define
1B4. read materials to whole
class
1C1. observations
1C2. predictions/
hypotheses
1C3. procedures
1C4. data
1C5. science (concept,
vocab words, mechanics
of science, etc…)
1D1. with evidence or
examples
1D2. does not apply
2. Through ______A______,
the teacher ______B_______
science ______C______.
2A1. direct instruction
questioning:
2A2a. rhetorical
2A2b. interactive
2B1. introduces or provides
an overview of
2B2. reviews/summarizes
2B3. demonstrates
2B4. collects
2B5. compares/contrasts
2B6. clarifies
2C1. concepts(idea)
2C2. procedures(activity)
2C3. tools/equipment
2C4. investigation
(science experiment)
2C5. problem
2C6. goal
2C7. -related safety issues
2C8 data (unknown):
2C8a. differences in
2C8b. relationships
between
2C8c. quality of
2C8d. analysis of
2C8e. synthesis of
2C8f. evaluation of
2C9. vocabulary words
2D1. new information
2D2. previously
learned information
2D3. investigation
2D4. unit
2D5. does not apply
60
1. Teacher directs student(s) _____A_____ to _____B______
_____C_____.
1. Teacher directs student(s) individually (1A1) to discuss (1B2)
procedures (1C3).
61
Inquiry Science Observation
Recording Sheet
• Notes the start time of when the “broader”
and “other activities” a teacher is engaged
in begins
• Allows for comments to facilitate the
reconciliation process
• Designed to record multiple activities
occurring at the same time
62
Broader and Other Activities
– The broader activity captures the larger activity
occurring:
Through interactive questioning (2A2b), the teacher
introduces or provides an overview of (2B1) science
investigation (2C4).
– The other activities capture the details occurring
within the broader activity:
Teacher questions students through clarifying (3A1a)
questioning and responds to student comment (3B1)
by repeating (3C2) the comment and probing further
(3C6).
63
64
Reconciling Codes
• Two coders are paired to compare their
independent codings, identifying any differences
between them.
• Differences are then discussed and each coder
provides a rationale for his or her selected code.
• When necessary, a review of the DVD is
conducted by the paired coders.
• Coding differences are discussed until
consensus is reached between coders.
65
Inquiry Science Observation
Reconciling Code Sheet
66
Challenges
• greatest challenges:
– establishing universal definitions for terms and
phrases
• What constitutes a “procedure”?
• What is the difference between a science “term” and a
science “concept”?
– standardizing the recording of activity strings using
the Recording Sheet
• broader verses other activities
– focusing on the teacher, rather than on the students
67
What Next?
• To date we have:
– videotaped 15 teachers;
– 140 videotape cassettes have been digitized
to DVDs;
– 116 have been quality-checked for audio
clarity and teacher visibility; and
– 8 coders are trained and ready to begin
coding!
68
Coding Teachers in
Science Classrooms using the
Inquiry Science Observation Guide
Alice K. H. Taum
Curriculum Research & Development Group
University of Hawai‘i at Mānoa
Developing Student Outcome
Measures Frameworks
Carlos C. Ayala
Sonoma State University
Scaling Up:
Student Measures
The Buck Stops Here
Carlos Ayala
Let’s Talk Charge
Develop student assessment instruments in
order to tease out differences between
professional development models.
2 Session Pre Test
3 Session Post Test Suite
Where we expect differences
•
•
•
•
•
Student’s content knowledge
Student’s science inquiry skills
Student’s views of the nature of science
Student’s efficacy toward science
Student’s motivation to learn science
73
Final Assessment Suite
• Pre-test
– 30 item multiple-choice/ short answer test (α= .86)
– Attitudinal Survey
• Post-test
– 30 item multiple-choice/ short answer test
– Attitudinal survey
– Mānoa River performance assessment
74
Targets
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
2 Buoyancy
Density
1 Density Of Liquids
Density Of Gases
Depth Of Sinking
Displacement
1 Temperature vs. Density
Mass
1 Water Cycle
Accumulation (Hydrology)
1 Condensation (Hydrology)
Absorption (Hydrology)
Dew Point
Evaporation (Hydrology)
Ground Water
Precipitation (Hydrology)
Percolation (Hydrology)
1 Radiation (Hydrology)
Transpiration (Hydrology)
Transportation (Hydrology)
Water And Soil (Capillary)
1 Weather
Atmosphere
1 Climate
Currents, Air
Fog, Rain, Dew, Clouds
Knowledge
Type
schematic
declarative
declarative
declarative
declarative
declarative
declarative
declarative
schematic
declarative
declarative
declarative
declarative
declarative
declarative
declarative
declarative
declarative
declarative
declarative
declarative
schematic
declarative
declarative
declarative
declarative
MC
CR
PA
POE
51
13
11
1
1
Concept
Map
12
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
75
Knowledge Type Framework
Type of
Knowledge
Definition
Examples
Prompt
Declarative
Knowing that
Concepts & facts
“What is percolation?”
Procedural
Knowing how
Actions, steps,
& procedures
Schematic
Knowing why
Principles &
mental models
“How do measure how much
water soil will hold?”
“How does the water cycle
work?”
76
77
Content Validity
•
•
•
•
•
•
Reviewed curriculum and created content matrices
Curriculum developers parsed content down
Linked matrices to materials
Created test linking items to matrices
Piloted assessments
Talk alouds
78
Student Science Inquiry
• As students become more engaged in the FAST
curriculum and the teacher more fully implements the
curriculum, students will be more proficient at
Science Inquiry.
– Design and conduct a scientific investigation
– Use appropriate tools to gather, analyze and interpret data
– Develop descriptions, explanations predictions and models
using evidence
• Targets based on
– Duschl’s Transformations in Three Domains
– Pottenger’s Deductive Explanatory Inquiry
NRC. (1996). National Science Education Standards . Washington D.C.: National
Academy of the Sciences.
79
Student Science Inquiry
Transformation 1
Data to Evidence: Deciding
if the data are evidence,
irrelevant and/or
problematic.
Reformulation: From
explanations to new
questions: Deciding what
next questions to ask and what
new data are needed.
Transformation 2: Evidence
to patterns or models
decisions about selecting
tools for identifying patterns
or models
Transformation 3:
Patterns and models to
explanations. Deciding how
the patterns or models lead to
explanations.
80
81
Nature of Science
• As students become more engaged in the
FAST curriculum and the teacher more fully
implements the curriculum, students will
understand that
–
–
–
–
anyone can be a scientist.
science knowledge is useful.
science knowledge builds over time.
science is creative.
Lederman, Abd-El-Khalick, Bell and Schwartz (2002) Views of Nature of Science
Questionnaire; Toward Valid and Meaningful Assessments of Learner’ Conceptions of
the Nature of Science.
82
Self Efficacy
• As students become more engaged in the FAST
curriculum, the greater control they will feel towards
science, science investigations and science
knowledge.
– I can make accurate measurements during a science
investigation.
– I can make appropriate predictions about what will happen
during a science investigation.
Britner, S. and Pajares F., (2001) Self-Efficacy Beliefs, Motivation, Race and Gender in
Middle School Science.
83
Motivation
• As students become more engaged in the FAST
curriculum, patterns of motivation may change.
• Dweck Groups (goals and epistemic beliefs)
– Mastery Orientation
– Ego Orientation
– Helpless Orientation
Haydel, A., & Roser, R. (2001). On the links between students' motivation patterns and
their perceptions of, beliefs about and performance on different types of science
achievement, Multidimensional Approach to Achievement
84
Next Steps
•
•
•
•
760 Tests and Surveys completed
Collect post test data
Run analyses
Provide results to group
85
Developing Student Outcome
Measures Frameworks
Carlos C. Ayala
Sonoma State University
The Differential Effects of
Two Versions of Professional
Development on Teachers’
Self-Efficacy to Implement
Inquiry-Based Science
Brian Lawton
University of Hawai‘i at Mānoa
Purposes
•
•
•
Effects of both versions of PD institutes
on teacher self-efficacy to implement
program
Differences between the two versions
Factors associated with self-efficacy
change
88
Data Collection
•
•
Qualitative and quantitative methods used
Self-efficacy scale
–
•
Focus groups
–
–
•
Developed to facilitate focus group discussion
Conducted immediately following the institutes
Provide in-depth information about changes in self-efficacy
Five stages of inquiry science
1.
2.
3.
4.
5.
•
Introducing new science investigations
Facilitating valid experimental design
Facilitating the investigation process
Constructing meaning
Linking knowledge to new situations
Seven teachers from the 10-day and seven teachers
from the 5-day participated in the study
89
Data Collection (cont.)
Self-Efficacy Scale
1. STAGE: Introducing new science investigations - my ability to introduce
students to new science investigations by reviewing and tying in previous
work.
Now
Low
ability
1
2
3
4
5
6
7
8
9
10
High
ability
10
High
ability
Before
Low
ability
1
2
3
4
5
6
7
8
9
90
Data Analysis
• Only presenting focus group results
• Analyzed for changes in self-efficacy
– Two categories:
• Statements implying an increase in self-efficacy
• Statements implying no change in self-efficacy
• Categories analyzed to identify the influencing
factors
• The statements and factors were quantified
91
Results, Influencing Factors
• Overall seven factors
were identified as
influencing self-efficacy.
Factors influencing increased
self-efficacy
•Content/pedagogical knowledge1
•Program Structure1
1
Factors identified as most
influential in increasing
self-efficacy
2 Factor
identified as most
associated with no
change in self-efficacy
•Professional Collaboration
•Experience
Factors influencing no change
in self-efficacy
•Practice/feedback2
•Student characteristics
•Experience
•Time issues
92
Results: Changes in Self-Efficacy
Changes in self-efficacy
10-day
version
5-day
version
Statements identified as
increases in self-efficacy
11
8
Statements identified as no
change in self-efficacy
6
9
93
Discussion
• Group increases in self-efficacy
• Changes in self-efficacy between the
groups
• Groups perceived gains in content and
pedagogical knowledge
• Importance of 5-day follow-on support
94
Further Study
• A study comparing the groups after the 5day group has received all their training.
95
Thanks!
96
The Differential Effects of
Two Versions of Professional
Development on Teachers’
Self-Efficacy to Implement
Inquiry-Based Science
Brian Lawton
University of Hawai‘i at Mānoa
Comparing the Effects of Two Versions
of Professional Development
on Science Curriculum
Implementation and Scaling-Up
Session 43.030
April 13, 2005
American Educational Research Association Annual Meeting
Montreal, Canada
Questions and Comments
• Paul Brandon - Overview
• Mary Gray and Truc Nguyen - Description of PD
• Paul Brandon - Log and questionnaire development
and questionnaire validation.
• Alice Taum –Observation guide development
• Carlos Ayala - Student assessment development
• Brian Lawton - Comparison of early effects of the
two versions of PD
Paper, Presentation and Contact Info available at:
http://hisii.hawaii.edu/SCUP/research/
99