Transcript Slide 1

I-RtI Network
Fidelity of Tier 2 Services
March, 2014
Facilitated/Presented by:
Insert name(s) here
The Illinois RtI Network is a State Personnel Development Grant (SPDG) project of the Illinois State Board of
Education. All funding (100%) is from federal sources.
The contents of this presentation were developed under a grant from the U.S. Department of Education, #H325A100005-12.
However, those contents do not necessarily represent the policy of the U.S. Department of Education, and you should not
assume endorsement by the Federal Government. (OSEP Project Officer: Grace Zamora Durán)
Making
What
Check-in
connections
Applying
Review of
February
One of the best
ways to remember
something is to
test yourself.
February’s
Fidelity
Checklist
Action Plans
Outcomes
for Today
Participant Agenda for March 2014 EC Meeting
Name:_________________________________ Date:________________
Meeting Content
1. Review of Feb EC meeting
content & field work
2. Key elements of fidelity
EC Fidelity Checklist Items
1-14
District RtI/MTSS
(1,2,3,5,8, 13 & 14 REQUIRED) manual
District RtI/MTSS
manual
3. Fidelity-curriculum
4. Fidelity-instruction
5. Fidelity-assessment
Extension Activity
Coaching Needs
1. Additional training
2. Additional coaching to apply this content
3. On-Site modeling
4. Assistance with problem solving barriers
5. Additional resources
15. District RTI/MTSS Manual:
District approach to
treatment fidelity for Tier 2.
District RtI/MTSS
manual
District RtI/MTSS
manual
District RtI/MTSS
manual
6. Fidelity-processes
District RtI/MTSS
manual
7. When to measure fidelity
more frequently
8. What level of fidelity to
expect
9. Systems issues
District RtI/MTSS
manual
District RtI/MTSS
manual
District RtI/MTSS
manual
Other Needs
I plan to communicate my coaching and technical assistance needs to my Lead Coach by:
Review
PreMeeting
Survey
Results
Fidelity of RtI/MTSS
• Fidelity is important
across all three tiers
• Each level has distinct
considerations for
fidelity
Adapted From National Center on Response to Intervention
8
How is fidelity at Tier 1
different and how is it
similar to fidelity for Tier
2?
Discuss at your table and as a group.
I-RtI Network
FIDELITY OF TIER 2 SERVICES
EC Fidelity Checklist
RTI/MTSS Manual
Item # 15:
District RTI/MTSS Manual: District
approach to treatment fidelity for Tier 2
“Inspect what you expect.”
I-RtI Network
WHY IS TREATMENT
FIDELITY IMPORTANT?
High Fidelity
Increased program
credibility
Highly Motivated Staff
More consistent and better
student outcomes
From NRCLD Fidelity of Implementation
IDEA and Appropriate
Instruction
• “In making a determination of eligibility under
paragraph (4)(A), a child shall not be
determined to be a child with a disability if the
determinant factor for such determination is–
(A) lack of appropriate instruction in reading,
including the essential components of reading instruction; (B) lack of instruction in math;
or (C) limited English proficiency [SEC
614.(b)(5)].”
Fidelity…
The ultimate goal is STUDENT SUCCESS.
Always examine
student
performance
data along with
fidelity data
I-RtI Network
BACKGROUND
INFORMATION AND WAYS TO
THINK ABOUT FIDELITY
Fidelity
Curriculum
• Are we adhering to the curriculum as
planned?
Instruction
• Are we adhering to instructional
strategies/routines as planned?
Assessment
• Are we administering & scoring
assessments reliably?
Process
• Are we adhering to the process as
planned?
SASED, 2013
Remember, we are examining
fidelity with lens of Tier 2
Measuring Fidelity of
Instruction
• Direct
– Observation checklist
• Indirect
– Permanent products
– Self-report
– Rating scales
– Interviews
Tools to Measure
•
•
•
•
•
•
•
Teacher log
Teacher report
Rating scale
Direct observation
Walk-through
Permanent product
Teacher evaluation instrument (Danielson or other)
Research on HOW to Measure
Fidelity
• Odom et al. (2010) compared structural
(objective), process (quality of delivery), and
multidimensional (both) measures of fidelity.
• Literacy best predicted by process
• Math best predicted by structural and
multidimensional
• Social best predicted by process and
multidimensional
• Overall recommendation-multidimensional
Odom, S. L., Fleming, K., Diamond, K., Lieber, J., Hanson, M, Butera, G., …Marquis, J. (2010). Examining different forms of
implementation in early childhood curriculum research. Early Childhood Research Quarterly, 25, 314-328.
http://dx.doi.org/10.1016/j.ecresq.2010.03.001
I-RtI Network
WHAT ARE THE KEY
ELEMENTS OF FIDELITY?
Your thoughts…
• Discuss with others at your table and write down
what you see as the key components of fidelity, for
example, the amount of time the student actually
receives the intervention.
• Remember to think about considerations specific to
Tier 2.
• Share ideas with the group.
Five Elements of Fidelity
Student Engagement: How
engaged and involved are
the students in this
intervention or activity?
Program specificity: How
well is the intervention
defined and different from
other interventions?
Student Adherence
Engagement
Program
Specificity
Exposure
Adherence: How well do
we stick to the plan/
curriculum/assessment?
Exposure/Duration: How
often does a student
receive an intervention?
How long does an
intervention last?
Quality of
Delivery
Quality of Delivery: How well is the intervention, assessment,
or instruction delivered? Do you use good teaching practices?
(Dane & Schneider, 1998; Gresham et
al., 1993; O’Donnell, 2008)
Slide from National Center on Response to Intervention
Fidelity
Curriculum
• Are we adhering to the curriculum as
planned?
Instruction
• Are we adhering to instructional
strategies/routines as planned?
Assessment
• Are we administering & scoring
assessments reliably?
Process
• Are we adhering to the process as
planned?
SASED, 2013
I-RtI Network
HOW CAN WE MEASURE
FIDELITY RELATED TO
CURRICULA (WHAT WE
TEACH)?
Fidelity
Curriculum
• Are we adhering to the curriculum as
planned?
SASED, 2013
Curriculum
• How do you identify the curricular area(s) in
which students need more instruction?
• Does the need for intervention reflect a gap in
your curriculum?
• How does your Tier 2 support align with your
Tier 1 support?
Example—Tier 2 Curriculum
Fidelity
Example—Instructional Planning
Form
Tier 2 Fidelity
Tools and Logistics for Measuring Tier 2 Fidelity
Tools We Have to
Measure Fidelity
(Direct and Indirect)
How often we
measure fidelity?
Who measures
fidelity?
Next steps to address
fidelity
Curriculum
-what is taught, including
intended and
unintended outcomes,
content of instruction
Instruction
-how content is taught,
delivery
Assessment
-how assessments are
administered,
interpreted
Process
-how processes, such as
problem solving, are
implemented
CONSIDER FIVE ELEMENTS OF FIDELITY: ADHERENCE, EXPOSURE/DURATION, QUALITY OF DELIVERY, PROGRAM
SPECIFICITY, AND STUDENT ENGAGEMENT
I-RtI Network
HOW CAN WE MEASURE
FIDELITY RELATED TO
INSTRUCTION (HOW WE
TEACH)?
Fidelity
Curriculum
• Are we adhering to the curriculum as
planned?
Instruction
• Are we adhering to instructional
strategies/routines as planned?
SASED, 2013
Instruction
• What is the research or evidence base
behind the intervention/strategy being
used?
• Was ample PD and training provided before
implementing the intervention/strategy?
• Who is making sure the
intervention/strategy is taught as designed?
• Student attendance
Example—Fidelity of Instruction
Tools and Logistics for Measuring Tier 2 Fidelity
Tools We Have to
Measure Fidelity
(Direct and Indirect)
How often we
measure fidelity?
Who measures
fidelity?
Next steps to address
fidelity
Curriculum
-what is taught, including
intended and unintended
outcomes, content of
instruction
Instruction
-how content is taught,
delivery
Assessment
-how assessments are
administered, interpreted
Process
-how processes, such as
problem solving, are
implemented
CONSIDER FIVE ELEMENTS OF FIDELITY: ADHERENCE, EXPOSURE/DURATION, QUALITY OF DELIVERY, PROGRAM
SPECIFICITY, AND STUDENT ENGAGEMENT
I-RtI Network
HOW CAN WE MEASURE
FIDELITY RELATED TO
ASSESSMENT?
Fidelity
Curriculum
• Are we adhering to the curriculum as
planned?
Instruction
• Are we adhering to instructional
strategies/routines as planned?
Assessment
• Are we administering & scoring
assessments reliably?
SASED, 2013
Assessment
• Does it measure student growth in the
targeted area?
• Is it reliable and valid?
• Is data easily accessible and presented in an
easily understood format?
• Are the tests administered accurately?
Example—Fidelity of Assessment
Checklist for Using CBM Before, During, and After Initial Implementation
BEFORE
Step 1: Who will be using CBM? (Check all that apply)
_____Classroom(s) _____Grade(s) _____School(s) _____District
Step 2: Which CBM skills will be implemented? (Check all that apply)
Early Reading
_____ DIBELS
_____ LSF (letter sounds)
_____WIF (word identification)
Reading
Sample Assessment Fidelity Action
Plan
GOAL
ACTION STEPS
GOAL DATE
To work with the
leadership team to choose
tools for fidelity checks
that fit with the school’s
needs, staff preferences,
organizational practices,
mission, priorities, and
values.
To have a documentation
system in place when
planning for fidelity checks.
CONTINUED NEXT PAGE
From Fidelity of Implementation Tools/NCRTI 2010
Sample Assessment Fidelity Action
Plan
GOAL
ACTION STEPS
GOAL DATE
To determine and
document a specific
schedule and place for
fidelity assessments.
To develop a system to
observe teachers for
fidelity of implementation
on screening, progress
monitoring, and
intervention application.
To provide teachers a
timeline and rationale on
how often and when
observations will occur.
CONTINUED NEXT PAGE
From Fidelity of Implementation Tools/NCRTI 2010
Sample Assessment Fidelity Action
Plan
Existing Practices and Materials:
Needed Practices and Materials:
From Fidelity of Implementation Tools/NCRTI 2010
Tools and Logistics for Measuring Tier 2 Fidelity
Tools We Have to
Measure Fidelity
(Direct and Indirect)
How often we
measure fidelity?
Who measures
fidelity?
Next steps to address
fidelity
Curriculum
-what is taught, including
intended and unintended
outcomes, content of
instruction
Instruction
-how content is taught,
delivery
Assessment
-how assessments are
administered, interpreted
Process
-how processes, such as
problem solving, are
implemented
CONSIDER FIVE ELEMENTS OF FIDELITY: ADHERENCE, EXPOSURE/DURATION, QUALITY OF DELIVERY, PROGRAM
SPECIFICITY, AND STUDENT ENGAGEMENT
I-RtI Network
HOW CAN WE MEASURE
FIDELITY RELATED TO
PROCESSES?
Fidelity
Curriculum
• Are we adhering to the curriculum as
planned?
Instruction
• Are we adhering to instructional
strategies/routines as planned?
Assessment
• Are we administering & scoring
assessments reliably?
Process
• Are we adhering to the process as
planned?
SASED, 2013
Process
• Is there ample time in the schedule for
intervention and progress monitoring?
• What data will be used to make decisions?
• Are meetings regularly scheduled?
• How are decisions made?
• How do we assess if the intervention is
effective?
Directions: For each selected student, please use the scale provided to indicate the degree to
which each critical component of Response to Intervention (RTI) is present in permanent
products (e.g., meeting notes, psychoeducational reports) that resulted from the RTI process.
See the attached rubric for the criteria for determining the degree to which each critical
component is present.
Example—Fidelity of Process
Component
Problem Identification
1. One or more replacement behaviors were identified.
2. Data describing current and expected levels of
performance were collected.
3. A gap analysis was conducted to determine the
appropriate tier of intervention
Problem Analysis
4. Hypotheses were developed across multiple domains.
5. Hypotheses were developed to determine if the
student was not performing the replacement behavior
because of a performance and/or skill deficit.
6. Data were used to determine viable or active
hypotheses for why the replacement behavior was
not occurring.
Intervention Development and Implementation
7. Short- and long-term goals were clearly stated in
relation to the benchmarks.
8. Interventions were developed to address barriers
identified by verified hypotheses.
9. An intervention plan specifying the logistics of the
intervention(s) selected was provided.
10. Data were provided demonstrating that the
intervention plan was implemented with integrity.
11. Intervention support plan with personnel was
developed.
Program Evaluation/RTI
12. Criteria for positive RTI were defined
13. Progress-monitoring data were collected/scheduled.
1 = Present
2 = Partially
Present
3 = Absent
1
1
2
2
3
3
1
2
3
1
1
2
2
3
3
2
3
1
1
2
3
1
2
3
1
2
3
1
2
3
1
2
3
1
1
2
2
3
3
Evidence/Comments
Checklists
 This video highlights the
implementation of a fidelity
checklist in a surgery. The film
shows the importance of
having a way to confirm that
practices were followed.
 Despite findings suggesting
the benefits, there was initial
resistance to the checklists
from surgeons.
http://www.pbs.org/newshour/bb/health/jan-june10/gawande_02-08.html
51
Checklists
• Discuss at your table and share with the
group:
– What were some things that surprised you about
checklists?
– How could you use a checklist to improve some
aspect of your work?
Sources for Fidelity Checklists
• Examples in Fidelity of Implementation Within
a RtI Framework document pages 16-35
• Share other resources (some links I have used
in the past now seem to be broken)
Created by Dr. Atul Gawande and
downloaded from http://www.projectcheck.org/checklist-for-checklists.html
Checklists-Optional Activity
• Provide time to create a checklist that you
could use to improve RtI/MTSS Tier 2
implementation. Examples:
– Process for grade level data meetings
– Using decision rules
– Setting goals
– Documenting interventions and progress
• Share checklists with the group
Tools and Logistics for Measuring Tier 2 Fidelity
Tools We Have to
Measure Fidelity
(Direct and Indirect)
How often we
measure fidelity?
Who measures
fidelity?
Next steps to address
fidelity
Curriculum
-what is taught, including
intended and unintended
outcomes, content of
instruction
Instruction
-how content is taught,
delivery
Assessment
-how assessments are
administered, interpreted
Process
-how processes, such as
problem solving, are
implemented
CONSIDER FIVE ELEMENTS OF FIDELITY: ADHERENCE, EXPOSURE/DURATION, QUALITY OF DELIVERY, PROGRAM
SPECIFICITY, AND STUDENT ENGAGEMENT
Time to Reflect
One thing I learned during this section…
One thing I would like to have clarified
is…
One way I could apply this learning is…
Questions/Comments
I-RtI Network
WHEN SHOULD WE MEASURE
FIDELITY MORE
FREQUENTLY?
Factors to Consider for HOW
OFTEN to Measure Fidelity?
•
•
•
•
Teacher experience level
Teacher requests for support
Overall class performance
Degree to which special education referrals do
or do not decrease
• Important to set up a timeline at beginning of the
year:
– Demonstrates importance
– Sets expectations early
From NRCLD
Recommendations for when to
measure fidelity more
frequently?
• Implementing practice for the first time
• When student performance data are low
• If less than 70% of students are responding to
any given intervention
• During first month of school year
Example: Reverse Request for
Assistance Tool-PBIS
• See tool included in handouts
• To be used when students are not making
sufficient progress in a Tier 2 intervention
Example: When To Take A Closer
Look
Indicator
New
curriculum/ne
w intervention
Comparatively
low class
average;
increase in
number of at
risk students
Established
instruction/inte
rventions
Person
Teacher/Coach
Coach
Fidelity Tool(s)
Frequency
Feedback/Follo
w-Up
Coaching
Direct
observation
with checklist
2-3 times per
semester
Coaching
Direct
observation
with checklist
Student work
sample
Teacher log
1 time per
month
Peer support
Professional
development
weekly
Student data
Observation
checklist
From Fidelity of Implementation within an RtI Framework
Continued next page
Example: When To Take A Closer
Look
Indicator
New
administrator/T
eacher
Early second
semester score
slump
Person
Fidelity Tool(s)
Frequency
Student data
School data
“Walkthrough”
“Walkthrough”
Observation
with checklist
Videotaping
From Fidelity of Implementation within an RtI Framework
Feedback/Follo
w-Up
Another Example: When To Take
A Closer Look
Indicator
Method of
Evaluation
Frequency
Support System
Hire new staff
Direct observations
Ideally 3
times/year, once
early on
Pairing with
mentor/coach,
provide training in
curriculum program
Class average lower
than school
average, larger
number of students
identified as “at
risk”
Direct observations Same schedule as
to review data,
screening- ideally 3
review teacher logs, times per year
review any
supporting
evidence from
parents, and review
student work
samples
From NRCLD Fidelity of Implementation
Teacher works with
mentor coach to
problem solve,
identify areas of
strength and
weakness, and
provide training
opportunities
Continued next page
Another Example: When To Take
A Closer Look
Indicator
Method of
Evaluation
Frequency
Support System
Observation/evalua
tion highlights
deficiency in
instructional
methods
Follow-up
observations,
dialogues with
teachers, teacher
logs/self-reports
As needed
Identify problem to
either require
professional
development
and/or re-allocation
of resources
From NRCLD Fidelity of Implementation
I-RtI Network
WHAT LEVEL OF FIDELITY
SHOULD BE EXPECTED AND IS
IT EVER ACCEPTABLE TO ADAPT
INTERVENTIONS/STRATEGIES?
Is it OK to adapt
interventions/strategies?
• Discuss at your table:
– Do you think it is ever acceptable to adapt
evidence-based interventions and strategies?
– If so, when is it acceptable to make changes?
– When is it not acceptable to make changes?
– What specific information would you use to make
decisions?
– What factors would you consider?
Adapting
Interventions/Strategies
Read:
Program Fidelity and adaptation: Meeting local
needs without compromising program effectiveness
O’Connor, Small, & Cooney (2007)
As you read put a * next to items you want to
highlight, a ! Next to items that surprise you, and a
? next to items you have a question about. Discuss
at your table.
Adapting
Interventions/StrategiesOptional
Reading
Read excerpts from Harn, Parisi, and Stoolmiller (2013).
Balancing fidelity with flexibility and fit: What do we
really know about fidelity of implementation in schools?
As you read put a * next to items you want to highlight, a
! Next to items that surprise you, and a ? next to items
you have a question about. Discuss at your table.
Matching Intervention and Contextual Variables to Maximize Fidelity (pages
184-185)
Documenting the Active Ingredients (page 185)
What is Acceptable Fidelity (page 186 until end of page 187)
Time to Reflect
One thing I learned during this section…
One thing I would like to have clarified
is…
One way I could apply this learning is…
Questions/Comments
I-RtI Network
CHALLENGES WITH
FIDELITY—ADDRESSING
SYSTEM ISSUES
Looking at the big picture
(Lake Michigan from the Sky)
Addressing system issues
• Changes in climate vs. changes in culture
• Professional development-adequate dosage of
training, technical support, coaching
• Resource allocation
• Complexity, materials required, perceived
effectiveness, interventionists (from NRCLD)
Example—Changes in Climate
and Culture
Climate
Supportive
Not Supportive
Values
Beliefs
Traditions
Norms
Leadership
From Fidelity of Implementation within a RtI Framework
How might you use the
Changes in Climate and
Culture tool?
• Discuss
• Do you think the tool could be helpful in your
district?
• If so, how could it be used?
• With what team?
• When would it be completed?
• What are some examples of “supportive” and
“not supportive” practices?
Leadership, Leadership,
Leadership
Sample Building
Administration Fidelity Action
Plan
GOAL
ACTION STEPS
GOAL DATE
To work diligently to
cultivate an atmosphere
that is conducive to trying
new approaches.
To help the school
integrate new
programming within its
existing practices and
routines.
CONTINUED NEXT PAGE
From Fidelity of Implementation Tools/NCRTI 2010
Sample Building
Administration Fidelity Action
Plan
GOAL
ACTION STEPS
GOAL DATE
To establish a leadership
team to facilitate decisionmaking about training
needs, staff development,
intervention
implementation, and
implementation of
assessments.
To assess staff’s vies,
morale, trust, collegiality,
and disagreement
resolution regarding the
assessment of treatment
fidelity
CONTINUED NEXT PAGE
From Fidelity of Implementation Tools/NCRTI 2010
Sample Building
Administration Fidelity Action
Plan
GOAL
ACTION STEPS
GOAL DATE
To encourage shared
decision making among
staff members
To facilitate a shared vision
among staff members
To develop effective
mechanisms for
encouraging frequent and
open communication and
feedback on overall
implementation efforts
CONTINUED NEXT PAGE
From Fidelity of Implementation Tools/NCRTI 2010
Sample Building
Administration Fidelity Action
Plan
GOAL
ACTION STEPS
GOAL DATE
To help department teams
develop procedures to
enhance strategic planning
by establishing clear roles
and responsibilities for
expected task
accomplishments
To choose effective leaders
to set priorities, establish
consensus, offer
incentives, and manage the
fidelity process
To ensure continuous
professional development
For complete document, see Fidelity of Implementation Tools/NCRTI 2010
How might you use the
Building Administration
Fidelity Action Plan?
• Discuss
• Do you think the tool could be helpful in your
district?
• If so, how could it be used?
• With whom?
• When would it be completed?
Professional Development
Read and discuss excerpts from Harn, Parisi, and
Stoolmiller (2013). Balancing fidelity with flexibility
and fit: What do we really know about fidelity of
implementation in schools?
As you read put a * next to items you want to
highlight, a ! Next to items that surprise you, and a
? next to items you have a question about. Discuss
at your table.
Considerations for Supporting and Sustaining Quality Implementation
of Evidence-Based Practices with Appropriate Fidelity (pages 188-190)
How does Jim Knight’s
Partnership Principles
and Coaching Model
Support Fidelity?
See review on next slide and discuss
at your table and as a group.
Partnership Principles
•
•
•
•
•
•
•
Equality—decide together
Choice—provide options
Voice—value opinions of all
Reflection—what worked well, what didn’t
Dialogue—share ideas and learn from others
Praxis—consider application of ideas
Reciprocity—everyone benefits from
contributions of others
From Instructional coaching: A partnership approach to improving
instruction (2007)
Coaching Steps
•
•
•
•
•
•
•
•
Enroll-who to work with?
Identify-what to work on?
Explain-clear explanation
Model-you watch me
Observe-I watch you
Explore-collaborative exploration of data
Refine-make changes as indicated
Reflect-how did it work?
From Instructional coaching: A partnership approach to improving
instruction (2007)
Overcoming Barriers and Planning to
Support Implementation Fidelity
• Review the Overcoming Barriers and Planning
to Support Implementation Fidelity sections of
the handout included in participant handouts
• Discuss how these could be used in your
district
• Optional: provide time for participants to
begin completing the planning document
Key Ideas for Tier 2 Fidelity
1. Implementation integrity is critical to
RtI/MTSS.
2. Choose the right tool to measure
integrity.
3. Always examine student performance
data.
4. Determine fidelity of:
1.
2.
3.
4.
Curriculum
Instruction
Assessment
Process
Any Other Key Ideas from Today:
Ask participants to summarize.
Closing
Activities
On-Site
Coaching