Transcript Examining the effects of a highly rated science curriculum

Studying Fidelity of Implementation (FOI):
How FOI influenced SCALE-uP’s Theory
of Action for Middle School Science
Curriculum Materials
*SCALE-uP = Scaling up Curriculum for Achievement Learning and Equity
Project, a partnership between George Washington University
and Montgomery County Public Schools**
Sharon Lynch, PI
Co-PIs: Curtis Pyke, Joel Kuipers, Michael Szesze** & Bonnie HansenGrafton**
http://www.gwu.edu/~scale-up/
Prepared for Researchers Without Borders Webinar, May 26, 2010
FOI Research Group
• Carol O’Donnell
• Suzanne Merchlinski & MCPS evaluation staff
• Bonnie Hansen-Grafton
• Joelle Lastica
• Vasuki Rethinam
• Bill Watson
• Rob Ochsendorf
• Liz Hatchuel
• Annie Hansen
With special thanks to MCPS middle school science
teachers who participated in this study
Disclaimer
• Indebted to the Interagency Educational
Research Initiative (IERI) administered by the
NSF, for research funding for SCALE-uP (7
year research program).
• I am currently working at NSF as a Program
Director in EHR/DRL ( return to my position
as a professor at GWU in September).
• However, the ideas and opinions discussed
here are entirely my own and in no way
represent the those of NSF.
SCALE-uP/FOI Webinar:
Cautionary Tale w/ Happy Ending
• Background of SCALE-uP and initial Theory of Action
• Year 0 Pilot Study: Curriculum modifications are
tricky business!
• Year 1+: Comparison groups are (incredibly) handy
in developing FOI instruments and understanding the
study context
• Year 2 & 3: Comparison group and FOI evidence are
crucial for credible evidence of effectiveness
• Year 4: Putting it all together: How the Theory of
Action was changed by FOI evidence: FOI as
“process” and “structure” constructs for both teacher
and student
Background for SCALE-uP and
Initial Theory of Action
• In 1990’s, AAAS Project 2061 developed a
Curriculum Analysis to identify curriculum
materials likely to help students learn a target
idea (benchmark/standard).
• Curriculum Analysis relied on experts’
judgment of written curriculum materials.
• Two parts:
--Focused, accurate, coherent content
on a standard/benchmark
--Instructional strategies contained in
written curriculum materials
Project 2061 Instructional Strategies for
curriculum materials
1. Convey sense of purpose
2. Address student ideas and
misconceptions
3. Promote engagement with relevant
phenomena
4. Developing, using scientific ideas
5. Encourage student thinking
6. Encourage assessment of progress
7. Creating positive learning environment:
curiosity, all students
AAAS. Project 2061.
Background for SCALE-uP
• Project 2061 Curriculum Analysis had located
only 2 acceptable curriculum units in middle
school science.
• Units had been field-tested with small
numbers of students (no comparison groups).
• Note. More mathematics curriculum materials
had acceptable ratings and were field-tested
and studied and scaled.
Background for SCALE-uP and
Research Questions
If science curriculum materials having Project
2061 attributes were studied in a series of
large (N = ~ 2000) quasi-experiments using
carefully matched comparison groups:
• Would they be effective?
• Would they be equitable?
• Would there be a relationship between fidelity
of implementation to a unit and student
outcomes?
• Could the materials be scaled-up in this large
school district?
• How did they function in classroom (videoethnography)?
SCALE-uP’s Interventions:
3 Science Units with Coherent
Focused on Target Ideas
• State of Michigan’s Chemistry That Applies
(CTA) focuses on conservation of matter. 8th
graders, unit ~ 6 weeks long.
• GEMS Lawrence Hall of Science Real
Reasons for the Season (Seasons) focuses
on the reasons for the Earth’s seasons. 7th
graders, unit, ~ 3 weeks.
• ARIES Harvard Smithsonian Motion and
Forces (M&F) focuses on portions of
Newton’s Laws. 6th graders, unit ~ 6 weeks
long.
Curriculum Analysis: Instructional Strategies
● =Excellent, ◕=Very Good, ◒=Satisfactory, ◔=Fair ○=Poor
Chemistry
That
Applies
ARIES
GEMS
Motion &
Forces
Seasons
Macmillan/
McGraw Hill
○
○
NR
◔
Conveying lesson/activity purpose
◕
◒
●
◔
Justifying lesson/activity sequence
◒
◒
◒
○
◒
◒
◒
○
○
◒
○
NR
○
○
○
○
◒
○
◒
○
Instructional Category
I. Identifying a Sense of Purpose
Conveying Unit Purpose
II. Taking Account of Student Ideas
Attending to prerequisite knowledge and skills
Alerting teacher to commonly held ideas
Assisting teacher in identifying own students’
ideas
Addressing commonly held ideas
● =Excellent, ◕=Very Good, ◒=Satisfactory, ◔=Fair ○=Poor
ARIES
GEMS
Motion &
Forces
Seasons
Macmillan/
McGraw Hill
●
●
●
●
○
◒
○
○
Introducing terms meaningfully
●
Representing ideas effectively
◒
◕
●
◒
◒
○
○
◒
●
◒
○
◔
○
○
○
●
●
○
◒
○
○
○
●
○
○
○
○
Instructional Category
Chemistry
That Applies
III. Engaging Students with Relevant Phenomena
Providing a variety of phenomena
Providing vivid experiences
IV. Developing and Using Scientific Ideas
Demonstrating use of knowledge
Providing practice
V. Promoting Student Thinking about Phenomena,
Experiences, and Knowledge
Encouraging students to explain their ideas
Guiding student interpretation and reasoning
Encouraging students to think about what they’ve
SCALE-uP’s Outcome Measures
• Curriculum-independent measure for
each unit focusing on the unit’s target
idea.
• Assessments had good psychometric
properties and were developed using a
Project 2061 assessment system.
• Multiple choice and constructed
response items designed to be
maximally accessible to students of
varied language skills.
Questions?
Background for SCALE-uP and
Initial Theory of Action, c. 2001
• Curriculum units highly rated on Curriculum Analysis
could be effective overall because:
– each focused coherently on one big idea/standard/
benchmark
– each had a carefully planned sequence of
activities, and
– each contained identified instructional strategies
leading students to construct understanding of one
target idea/benchmark/standard.
Big Question: Would they be equitable?
Assumption: “Business as usual” comparison
classrooms would be less focused, rely more on
textbooks and worksheets, and provide less time
for guided inquiry and lab work.
“Typical” Theory of Action
Teacher
Curriculum
Materials
Student
Outcomes
Fidelity of implementation
Mowbray, Holter, Teague & Bybee, 2003
• “Fidelity of implementation is the extent to which the
delivery of an intervention adheres to the original
program theory behind its development; it confirms
that the implementation of the independent variable
in outcome research occurred as planned…
• …(and involves) the dynamic nature of fidelity
criteria, appropriate validation and statistical analysis
methods, the inclusion of structure and process
criteria in fidelity assessment and the role of
program theory in deciding on the balance between
adaptation versus exact replication of model
programs.”
SCALE-uP Theory of Action
c. 2001
Teacher FOI
Instructional
Strategies
Teacher
Curriculum
Materials
Student
Outcomes
SCALE-uP Theory of Action
c. 2001
• Teachers would need to locate and enact the
instructional strategies embedded in the curriculum
unit (identified via the Project 2061 Curriculum
Analysis).
• Doing this well would be to “implement with fidelity”.
• Hypothesis: The better the implementation of the
unit’s embedded instructional strategies in a
classroom, the higher the student outcomes.
• All SCALE-uP needed to do to create a classroom
observation instrument that captured teachers’
implementation of the strategies.
• This would become Instructional Strategies
Classroom Observation Protocol (more on this later).
• Simple!
Questions?
On to Fidelity of Implementation
(FOI)!
Year 0 (Pilot Study) Results:
Chemistry That Applies (CTA)
with Modifications Encouraged
• CTA’s results showed effect sizes = .52.
• Results disaggregated for subgroups of students
showed that no students disadvantaged by CTA.
• Teachers were asked to modify CTA for diverse
learners and record modifications.
• Virtually none did! This raised questions about
whether to modify CTA in future. Teacher beliefs
about the unit were mixed.
• Co-PI Szesze wanted to be sure units were
unambiguously effective, or not.
• Decision to “implement with fidelity” in the future
studies .
• Teachers and researchers drew up fidelity guidelines
together.
Lesson Learned :Modifying
Curriculum Unit is Tricky Business!
• If an intervention does not have solid
evidence of efficacy/effectiveness, then
modifications muddy the water; CTA and the
other units were “unproven” interventions.
• Therefore, their critical components could not
be “known”, but assumed.
• In retrospect, this was a good decision for
SCALE-uP; capturing modifications while
trying to establish if a unit “worked” would not
be credible in this school district context.
Iterative Process of Identifying Critical
Attributes, Measuring Attributes, and
Looking for Relationships between FOI
and Outcomes
Identify Critical
Attributes of
Intervention
FOI
measure
Reconsider
Critical
Attributes
Outcomes
Mowbray, C., Holter, M. C., Teague, G. B., & Bybee, D. (2003). Fidelity criteria: Development,
measurement, and validation. American Journal of Evaluation, 24(3),315-340.
Questions?
On to Year 1
Replication of CTA
Building the first FOI Instrument
Yr. 1 Replication of CTA and
Develop ISCOP/FOI Process
• Given Theory of Action (the more teachers
implemented Project 2061 instructional strategies
embedded in curriculum units, the higher student
outcomes), we began to develop an instrument that
could capture fidelity to identified Project 2061
instructional strategies.
• A “generic” instrument for all 3 units studied, but the
units had been carefully vetted and had much in
common.
• ISCOP (Instructional Strategies Classroom
Observation Instrument), the first FOI measure was
born, and developed over the next 4 years.
Effect Sizes: CTA (Year 1)
Years 1, 2, 3, 4:
Test ISCOP in Treatment and
Comparison Classrooms
• Was the ISCOP capturing fidelity to instructional
strategies in a way that discriminated between
Treatment and Comparison classrooms?
• ISCOP did not discriminate very well, so it was
refined and refined and refined.
• Was ISCOP a “bad” instrument? Or were Treatment
and Comparison classrooms similar for Instructional
Strategies?
• If Treatment and Comparison classrooms were
similar in strategies, should the Theory of Action be
revisited?
Lesson Learned: Comparison Classrooms
Invaluable as Counterfactuals
• ISCOP data suggested that more complexity
than assumed in the Theory of Action.
• Measuring FOI relying solely with ISCOP
might not answer FOI research question if
comparison classroom data were taken into
consideration.
• Generic measures of instructional
strategies/FOI process are notoriously hard to
develop—valid and reliable?
Questions?
On to Years 2 and 3 with a new unit,
Real Reasons for the Seasons
(Seasons)
Year 2 and 3 Results:
Comparison group outscores
Seasons group: How to account
for this?
• Seasons studied in 7th grade
classrooms (N ~ 2000 students and ~
40 classrooms).
• Year 2 and 3 results: Comparison group
has higher outcome than Seasons on
curriculum independent measure (ES =
-.36, -.18, respectively).
Comparison Classrooms Invaluable as
Counterfactuals
• Comparison classroom teachers surveyed
and interviewed.
• Data showed that Comparison classrooms
were:
-Non-traditional
-Variety of curriculum materials used,
including other inquiry units
-Focused on the target idea
-Equal Duration of Treatment and
Comparison units
-Comparison and Treatment teacher
characteristics similar.
Seasons Replication in Year 3:
Focus on FOI
• Use ISCOP in Treatment and Comparison
classrooms.
• Develop a new Lesson Flow FOI process
measure that gauged Teacher, StudentGroup or Individual Centeredness of
classrooms based on hunch that StudentGroup Centeredness was important to
student construction of science ideas in
groups.
• Videotaped a Seasons and Comparison
classroom.
• Interviewed and surveyed Seasons and
Comparison Teachers again.
Lesson Flow Classroom Observation
Instrument
Lesson Flow Classroom Observation Protocol (LFCOP)
Effect Sizes: Seasons (Year 3)
Overall ES = -.18
Year 3 FOI Results
• Only 3 items on ISCOP showed significant
differences between Seasons and Comparison
classrooms, and 2 of 3 were observed more often in
Comparison classrooms.
• Lesson Flow: Seasons classrooms were more
teacher-centered than Comparison:
Teacher-Centeredness
Seasons =
71 % of time
Comparison =
58 % of time
• Video-data backed this up.
• Emerging Conjecture: Students need time in to work
and talk in groups to develop their ideas and
Seasons allowed less time for this.
SCALE-uP Theory of Action
c. 2006
Teacher FOI
Instructional
Strategies
Teacher
Student
Outcomes
Curriculum
Materials
Student FOI
Student Group
Centeredness
Changing Theory of Action and
Approach to FOI c. 2006
• Focusing FOI entirely on teacher in a guided
inquiry unit may be simplistic.
• Student agency implicated in FOI constructs.
• But Lesson Flow is controlled by the teacher
who provides students with the time/space to
work in groups; students can choose to do
the science work, or not.
• Lesson Flow is not Time On Task.
Seasons Unit and FOI:
The Research Re-visited
• Seasons was designed to be a
supplementary unit.
• Developers seemed not to understand the
implications of this when agreement was
made to study effectiveness/FOI.
• SCALE-uP researchers did not appreciate the
differences in philosophy of Seasons with
research design.
• Seasons was a bad match for this study.
• Revisions made to Seasons based upon
feedback from MCPS teachers.
Questions?
On to Year 4 and Motion and Forces
(M&F)
Note this is the third curriculum unit studied,
not to be confused with the first two, and was
implemented in 6th grade classrooms.
M&F Results in Years 2 and 3:
Underwhelming
• Year 2 ES = +.10
• Year 3 ES = -.06
• FOI had not been emphasized, so there was
little information to explain lackluster
outcomes.
• However, we learned that students had not
been issued M&F student Journals in Years 2
and 3. Was this an important, overlooked
FOI issue?
Year 4 Replication of M&F:
Focus on FOI
• Replicated quasi-experiment in 10 new
schools (N ~ 2000 students).
• Teachers asked to focus on FOI.
• Wonderful teachers who understood the
study goal: study the impact of M&F.
FOI in Year 4 with M&F
FOI included:
• ISCOP (Process FOI)
• Lesson Flow (Process FOI)
• Adherence to unit (ACOP), a new instrument
that measured close adherence to M&F’s
structure (Structure FOI)
• Teacher Interviews/Surveys
• Teacher Logs
• Student Journal Entries, # of responses to
Journal (Structure FOI)
• Student survey about self-reporting use of
instructional strategies (Process FOI)
Overall Results for M&F in Year 4
• Student Level results using traditional
ANOVA, ES = .23
• Classroom Level results using HLM,
ES = .56 (Rethinam, Lynch, & Pyke,
2008)
Effect Sizes for subgroups of
students: M&F (Year 4)
ISCOP Strategies: Means and
Correlations with Outcomes for M&F
Instructional Criterion
Mean (Scale = 0 - 3)
Correlation with Outcomes
Treatment
Comparison
Treatment
Comparison
Conveying lesson/activity purpose
0.65
0.58
.57**
-.28
Justifying lesson/activity purpose
.46
.58
.56**
-.40
2.40
1.38
-.05
-.52**
Providing variety of phenomena
1.71
1.50
.44*
-.23
Providing vivid experiences
2.75
2.46
.13
.08
Introducing terms meaningfully
2.48
2.38
.60**
-.09
Representing ideas effectively
1.46
1.13
.16
-.01
2.13
2.25
.62**
-.20
Identifying a Sense of Purpose
Taking Account of Student Ideas
Assisting teacher in identifying own
students’ ideas
Engaging Students with Relevant Phenomena
Developing and Using Scientific Ideas
Promoting Student Thinking about
Phenomena, Experiences, Knowledge
Encouraging students to explain their
ideas
ISCOP
• This “generic” instrument for measuring FOI
instructional strategies did not obviously
distinguish between M&F and Comparison
classrooms.
• But some items were correlated with student
outcomes for M&F classrooms.
• O’Donnell dissertation elegantly teased out
which instructional strategies seemed to
matter for higher student outcomes.
• ISCOP needs more work on validity and
reliability--OR SCALE-uP Theory of Change
needs refinement—Project 2061 Curriculum
Analysis needs refinement?
Lesson Flow for Instruction for
units on Motion and Force
Group
Group
Teacher
Individual
Teacher
Individual
M&F
Comparison
Lesson Flow
• M&F classrooms provided more time for
students to work in groups and
individually than Comparison
classrooms.
• Students construct meaning in groups,
consistent with notion of community of
practice and situated cognition.
Adhering to M&F Lesson
Components (ACOP)
• M&F teachers adhered to M&F unit > 80% of
time.
• ACOP did not predict outcomes because the
range was narrow; this was a good thing for
this study because high fidelity in this
measure of “structure” is credible and
strong—good face validity for FOI, and highly
reliable—the unit was well-implemented.
• Teachers adhered to the unit for this study, to
ascertain impact of M&F.
• Thank you teachers.
Results: Teacher FOI Structure
Adherence Classroom
Observations Protocol ACOP Item
n
30
%Not
implemented
--
%Somewhat
implemented
--
%Mostly
implemented
53.3
%Fully
implemented
46.7
Teacher provides students with
materials
Teacher provides opportunity
to write responses
Teacher discuss with students
responses to questions
Teacher provides opportunity
to set up equipment
Teacher provides opportunity
to record and/or discuss
predictions
Teacher provides opportunity
to conduct investigation
Teacher provides opportunity
to reflect and write
Interpreting results and
discussion in writing
Interpreting results and
discussion verbally
Teacher Guide Additions
30
--
--
10.0
90.0
30
13.3
3.3
3.3
80.0
30
10.0
--
33.3
56.7
8
12.5
--
--
87.5
30
--
3.3
6.7
90.0
20
15.0
--
--
85.0
30
--
--
13.3
86.7
30
13.3
6.7
20.0
60.0
24
41.7
41.7
8.3
8.3
Student Journals
• M&F students completed their journal
responses ~ 80% of time.
• Rates of journal question completion
predicted classroom outcomes.
Unpublished Year 4 HLM Results for
M&F: ISCOP, ACOP, Lesson Flow,
Student Journals
• HLM analysis found one classroom-level
factor that predicted student outcomes:
--Amount of Student-GroupCenteredness (Lesson Flow)
• and one student level factor
--Student Journal Completion
•
Thanks to Dr. Jaewa Choi of GWU for this data analysis.
SCALE-uP Theory of Action
c. 2010
ISCOP
Teacher
Instructional
Strategies
Process
FOI
Curriculum
Materials
Teacher
Process
FOI
Lesson Flow
Student Group
Centered
Instruction
Teacher
Adherence
to Lesson
Structure
ACOP
Structure
FOI
Student
Outcomes
Student
Adherence
to Lessons
Structure
FOI
Student Journal
SCALE-uP Theory of Action
c. 2010
ISCOP
Teacher
Teacher
Instructional
Strategies
Curriculum
Materials
Student Group
Centered
Instruction
Lesson Flow
Teacher
Adherence
to Lesson
Structure
ACOP
Student
Outcomes
Student
Adherence
to Lessons
Student Journal
SCALE-uP Theory of Action
c. 2010
ISCOP
Teacher
Instructional
Strategies
Teacher
Adherence
to Lesson
Structure
ACOP
Teacher
Curriculum
Materials
Student
Outcomes
Students
Student Group
Centered
Instruction
Lesson Flow
Student
Adherence
to Lessons
Student Journal
SCALE-uP Theory of Action
c. 2010
ACOP
Teacher
Curriculum
Materials
Teacher
Instructional
Strategies
Teacher
Adherence
to Lesson
Structure
ISCOP
Student
Outcomes
Students
Lesson Flow
Student Group
Centered
Instruction
Student
Adherence
to Lessons
Student Journal
Questions?
Summary Findings
• Score Card: CTA and M&F seemed to be effective
and equitable in this school system, Seasons did not
seem to be effective.
• Theory of Action: Changed from looking at one
relatively “generic” measure of teacher process
fidelity to multiple measures of FOI adding student
FOI and measures of FOI structure, and developing a
better Theory of Action.
• Lesson Flow and Student Journal response best
predicted student outcomes; ISCOP analysis
illuminated most potent instructional strategies for
M&F.
FOI Lessons Learned
• Studying curriculum adaptations for an
“unproven” curriculum is a slippery slope
because critical ingredients are unknown.
• Study of FOI in Comparison groups provides
important counterfactual.
• This study shows potential for both teacher
and student FOI, as well as process and
structural approaches.
• Study of FOI should reveal more about critical
components of an intervention if measures of
each component shows a positive
relationship with student outcomes.
The End
Thanks to Researchers Without
Borders for sponsoring this Webinar.
References
•
•
American Association for the Advancement of Science (AAAS). (2003). Project 2061 middle grades
science textbooks: A Benchmarks-based evaluation. Retrieved June 1, 2004, from
http://www.project2061.org/tools/textbook/mgsci/index.htm.
Dusenbury, L., Brannigan, R., Falco, M., & Hansen, W. B. (2003). A review of research on fidelity
of implementation: Implications for drug abuse prevention in school settings. Health Education
Research Theory and Practice, 18(2), 237-256.
•
Harvard-Smithsonian Center for Astrophysics. (2001). ARIES: Exploring motion and forces: Speed,
acceleration, and friction. Watertown, MA: Charlesbridge Publishing
•
Kesidou, S., & Roseman, J.E. (2002). How well do middle school science programs measure up?
Findings from Project 2061’s curriculum review. Journal of Research in Science Teaching, 39(6), p.
522-549
•
Lastica, J.R., & O'Donnell, C.L. (2007, April). Considering the role of fidelity of implementation in
science education research: Fidelity as teacher and student adherence to structure. In C. O'Donnell
(Chair), Analyzing the relationship between Fidelity of Implementation (FOI) and student outcomes
in a quasi-experiment. Symposium conducted at the Annual Meeting of the American Educational
Research Association, Chicago, IL.
Lawrence Hall of Science. (2000).GEMS: The real reasons for seasons—Sun-Earth connections.
Berkeley: The Regents of the University of California.
•
Lynch, S. (2000). Equity and science education reform. Mahwah, NJ: Lawrence Erlbaum and
Associates.
•
Lynch, S., Kuipers, J.C., Pyke, C., & Szesze, M. (2005). Examining the effects of a highly rated
science curriculum unit on diverse students: Results from a planning grant. Journal of Research in
Science Teaching, 42(8), 912-946.
•
Lynch, S. & O’Donnell, C. (2005, April). The evolving definition, measurement, and
conceptualization of fidelity of implementation in scale-up of highly rated science curriculum units in
diverse middle schools. In S. Lynch (Chair), The role of fidelity of implementation in quasiexperimental and experimental research designs: Applications in four studies of innovative science
curriculum materials and diverse student populations. Symposium conducted at the Annual Meeting
of the American Educational Researchers Association, Montreal, Canada.
•
Lynch, S., O'Donnell, C., Hatchuel, E., & Rethinam, V. (2007, April). A model predicting student
outcomes in middle school science classrooms implementing a highly-rated science curriculum unit.
Paper presented at the Annual Meeting of the National Association for Research in Science
Teaching, New Orleans, LA.
•
Lynch, S., O'Donnell, C., Hatchuel, E., Rethinam, V., Merchlinsky, S., & Watson, W. (2006, April).
What’s up with the Comparison group?: How large quasi-experimental study of highly rated science
curriculum units came to grips with unexpected results. Paper presented at the Annual Meeting of the
American Educational Research Association, San Francisco, CA.
•
Lynch, S., Taymans, J. Watson, W., Ochsendorf, R., Pyke, C. & Szesze, M. (2007). Effectiveness of
a highly-rated science curriculum unit for students with disabilities in general education classrooms.
Exceptional Children, 73(2), 202-223.
•
Merchlinsky, S. & Hansen-Grafton, B. (2007, April). Considering the role of Fidelity of
Implementation (FOI) in science education research: Evaluation and science specialists' role in
collecting FOI data in a large school district. In C.L.
•
Michigan Department of Education. (1993). Chemistry That Applies. The State of Michigan.
•
Mowbray, C., Holter, M. C., Teague, G. B., & Bybee, D. (2003). Fidelity criteria: Development,
measurement, and validation. American Journal of Evaluation, 24(3),315-340.
•
O’Donnell, C. L. (2007). Fidelity of implementation to instructional strategies as a moderator of
curriculum unit effectiveness in a large-scale middle school science experiment. Dissertation
Abstracts International, 68(08). (UMI No. AAT 3276564)
•
O'Donnell, C.L., Lynch, S., Lastica, J., & Merchlinsky, S. (2007, April).
Analyzing the relationship between Fidelity of Implementation (FOI) and
student outcomes in a quasi-experiment. Symposium conducted at the Annual
Meeting of the American Educational Research Association, Chicago, IL.
•
O'Donnell, C.L., Lynch, S., Watson, W., & Rethinam, V. (2007, April).
Teacher and student Fidelity of Implementation (FOI) to process: Quality of
delivery and student responsiveness and relationships to classroom
achievement. In C.L. O'Donnell (Chair), Analyzing the relationship between
Fidelity of Implementation (FOI) and student outcomes in a quasi-experiment.
Symposium conducted at the Annual Meeting of the American Educational
Research Association, Chicago, IL.
•
O'Donnell, C. (2008). Defining, conceptualizing, and measuring fidelity of
implementation and its relationship to outcomes in K-12 curriculum
intervention research. Review of Educational Research, 78(1), 33-84.
•
O'Donnell, C., & Lynch, S. (2008, March). Fidelity of implementation to
instructional strategies as a moderator of science curriculum unit effectiveness.
Paper presented at the 2008 annual meeting of the American Educational
Research Association, New York, NY.
•
Rethinam, V., Pyke, C., & Lynch, S. (2008). Using multilevel analyses to
study the effectiveness of science curriculum materials. Evaluation and
Research in Education, 21(1), 18-42
•
Rethinam, V., Pyke, C., & Lynch, S. (in preparation.). Using Multilevel
Analyses to Study Individual and Classroom Factors in Science Curriculum
Effectiveness.
•
Songer, N. B., & Gotwals, A. W. (2005, April). Fidelity of implementation in
three sequential curricular units. In S. Lynch (Chair), “Fidelity of
implementation” in implementation and scale-up research designs:
Applications from four studies of innovative science curriculum materials and
diverse populations. Symposium conducted at the meeting of the Annual
Meeting of the American Educational Research Association. Montreal,
Canada.
•
Stern, L. & Ahlgren, A. (2002). Analysis of students’ assessments in middle
school curriculum materials: Aiming precisely at benchmarks and standards.
Journal of Research in Science Teaching, 39, 889-910.
•
Watson, W., Lynch, S., Rethinam, V., & O’Donnell, C. (2006, April).
Development of an instrument to measure student responsiveness to
implementation of science curriculum materials. Paper given at the annual
meeting of the National Association for Research in Science Teaching, April,
2006 , San Francisco.
•
Further acknowledgments of important contributions to this work on fidelity of
implementation made by Rob Ochsendorf, Aiyita Ruiz-Primo, Doug
Clement, Okhee Lee, Bruce Ward, Carolyn Walton, Theron Blakeslee, Andy
Anderson, Phyllis Blumenfeld.