Assessing Response to Intervention

A collaborative project between the Florida Department of Education and the University of South Florida FloridaRtI.usf.edu

Advance Organizer

Team Activity - Review Day Four Intervention Plan Why Monitor Progress?

Graphing Conventions Goal Setting Interpreting Graphs Decision Making Review of Problem-Solving Steps

Team Activity Review Day 4 Intervention Plan

1. How did the intervention plan your team wrote on Day 4 differ from intervention plans typically written at your school? How was the plan similar?

2. For which components of the intervention plan did your team provide the most descriptive and specific details?

3. For which components of the intervention plan could your team have provided additional detail to clarify what needed to happen for implementation to occur?

4. How has the writing of intervention plans changed since your team attended the Day 4 training? How have the plans remained the same?

Beliefs Survey

• • Your project ID is: Last 4 digits of SS# Last 2 digits of year of birth

Response to Intervention in Context

Identify the Problem Analyze the Problem Select/Design Intervention Implement Intervention Monitor Progress Evaluate Intervention Effectiveness

L J

Timeline

• • • • •

Why Monitor Progress?

Unless we monitor progress, we cannot determine the rate at which the gap is closing.

Continuous feedback improves instructional planning (formative assessment) Allows earlier decisions about what to do Increase time, decrease time, revisit problem-solving, etc.

Provides measure of intervention effects -Getting better, staying the same, or getting worse We don’t know the effectiveness of an intervention until we implement it and monitor progress Randy Allison, 2004

Why Monitor Progress?

• Provides clear idea of expectations of performance over time. • Student outcomes improve when performance is assessed regularly • • Allows visual comparison to a standard Data collection provides an objective data base for decision-making • • Know if learning is being enabled, and if so, under what conditions Continuous feedback on performance enhances motivation for many Randy Allison, 2004

Outcome of Monitoring: Diagnose Conditions that Enable Learning

• Identify conditions under which student desired learning and behavior is accelerated and you have determined what enables learning.

• Hi Lo If conditions are configured like X, Y, or Z, then…what benefits result for the student?

X Y

. ........

Time Z

.....

. .......

Randy Allison, 2004

Why Not Just Use Pre - Post Test Comparisons?

• • • • • We must measure and demonstrate more than improvement. We must show the rate necessary to attain benchmarks within a time frame.

May be unreliable because of small amount of data collected May be significant time lag between pre-test and post-test Not sensitive to small changes in direction of performance in a timely manner More difficult to analyze patterns of performance Randy Allison, 2004

Basics of CBM - Scientifically Based Progress Monitoring

• Research Efforts Led By

Stanley Deno

, Beginning in

1971

with Federal Funding in 1978 to Provide Viable Progress Monitoring toward IEP Goals • Almost

30 Years of Continuous Research

•

Mid 1990s

Witnessed Move to

Standard, High Quality Assessment Materials

•

A Number of Members of the CBM

“Family” including DIBELS M. Shinn

Why CBM…or any Other Measure?

• • • • • • Quick and inexpensive Linked to instruction and curriculum Frequently repeatable Sensitive to small increments of growth Reliable and valid Can be used for multiple purposes and to answer different questions Jenkins, Deno, and Mirkin

Academic Measures -CBM-

Reading Fluency Words read correctly Reading Comprehension Spelling Written Expression Math Computation Math Applications Correct words inserted when the seventh word is left out Correct words spelled and correct letter sequences Words written or correct word sequences written Correct digits written from math computation problems Correct digits written from math application problems From M.Shinn

Behavioral Measures

• • • • • • • • • • Office discipline reports Behavioral incidents Suspension/Detention records Observations Self-assessments Surveys Attendance data Teacher checklists Screening instruments Rating scales

Sam Digits correct per minute 6 5 12 11 10 9 8 7 4 3 2 1 1 2 3 4 5 6 7 8 Days 12

Why use graphs?

•

Teachers

are able to make sound decisions about the instruction being delivered to students based upon data, not guesswork •

Parents

are kept well informed about their child’s progress with specific information about how their child is responding to instruction. Parents may assist in making suggestions for instructional adjustments.

•

Students

know what is expected of them. They receive specific feedback about their performance along the way rather than only at the end of the marking period. Goal setting and progress monitoring are some of the most effective strategies to improve academic engaged time.

General Outcomes / Specific Skills

• The general outcome expectations for students are comprised of many requisite subskills. Both general outcomes and specific skills may be measured.

• What subskills would be necessary for:  Long division?

 Fluent reading?

 Compliance with teacher direction?

Skill equal increments

Graph Components

Instructional Change Line

Baseline Intervention (Group or Individual)

Goal Aim Line Trend Line Time equal increments

Skill

Keep increments consistent!

2 wks 2 wks

Time

Skill

Keep increments consistent!

In both conditions, the rate of skill acquisition is 1/week.

Dissimilar x-axis increments give the impression that learning rate is increased during second time interval 2 wks 2 wks

Time

Skill

Keep increments consistent!

Similar x-axis increments reveal true progress over time 2 wks 2 wks

Time

Goal Setting

• Set goals which are ambitious, but reasonable -- rate required to reach goal is 25-50% above typical student rate • Goal has two components Level of performance desired Time within which that level will be attained

Goal Setting

• • Measure difference between desired and current performance Divide by number of weeks • Compare to standard to determine reasonable & ambitious growth rate (an increase of 25-50% of typical rate)

Goal Setting

Desired - Current Number of Weeks 110 wcm - 60 wcm 20 weeks 50 wcm 20 weeks 2.5 wcm/week 2.0 wcm/week is typical Compare to rate for typical peers: in 25-50% range?

Goal Setting

To determine an increase of 25-50% For lower end of range - multiply typical rate X 1.25

For upper end of range - multiply typical rate X 1.50

Examples: If typical rate = 2.00 words correct per minute/week Ambitious range would be 2.5 - 3.0 words correct per minute/week If typical rate = 3.00 digits correct per minute/week Ambitious range would be 3.75 - 4.50 digits correct per minute/week

Goal Setting

If: 2.0 letter sounds correct per minute/week is typical What is ambitious range?

Goal is .75 digits correct per minute/week & .5 digits correct per minute/week is typical Is this in ambitious range?

Goal is 3.5 words correct per minute/week & 2.0 words correct per minute/week is typical Is this in ambitious range?

Example Growth Rates

• • • • • • • Realistic growth rates (words/ week) Grade 1 2.00 words Grade 2 1.50 words Grade 3 1.00 words Grade 4 .90 words Grade 5 .50 words Grade 6 .30 words • • • • • • • Ambitious growth rates (words/week) Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 3.00 words 2.00 words 1.50 words 1.10 words .80 words .65 words Fuchs, Fuchs, 1993

•

Level

Attend to:

•

Slope / Rate

•

Variability

Decision Rules: What is a “Good” Response to Intervention?

•

Positive Response

 Gap is closing  Can extrapolate point at which target student(s) will “come in range” of target--even if this is long range •

Questionable Response

 Rate at which gap is widening slows considerably, but gap is still widening  Gap stops widening but closure does not occur •

Poor Response

 Gap continues to widen with no change in rate.

Positive Response to Intervention

Expected Performance

Performance Fall Winter

Observed Performance

Spring

Positive Response to Intervention Performance

Expected Trajectory Observed Trajectory

Time

Decision Rules: What is a “Good” Response to Intervention?

•

Positive Response

 Gap is closing  Can extrapolate point at which target student(s) will “come in range” of target--even if this is long range •

Questionable Response

 Rate at which gap is widening slows considerably, but gap is still widening  Gap stops widening but closure does not occur •

Poor Response

 Gap continues to widen with no change in rate.

Questionable Response to Intervention

Expected Performance

Performance Fall Winter

Observed Performance

Spring

Questionable Response to Intervention Performance

Expected Trajectory Observed Trajectory

Time

Decision Rules: What is a “Good” Response to Intervention?

•

Positive Response

 Gap is closing  Can extrapolate point at which target student(s) will “come in range” of target--even if this is long range •

Questionable Response

 Rate at which gap is widening slows considerably, but gap is still widening  Gap stops widening but closure does not occur •

Poor Response

 Gap continues to widen with no change in rate.

Poor Response to Intervention

Expected Performance

Performance Fall Winter

Observed Performance

Spring

Poor Response to Intervention Performance

Expected Trajectory Observed Trajectory

Time

Response to Intervention Performance

Expected Trajectory Positive Questionable Poor Observed Trajectory

Time

Decisions What to do if RtI is:

•

Positive

• Continue intervention with current goal • Continue intervention with goal increased • Fade intervention to determine if student(s) have acquired functional independence.

Decisions What to do if RtI is:

•

Questionable

 Was intervention implemented as intended?

• If no - employ strategies to increase implementation integrity • If yes  Increase intensity of current intervention for a short period of time and assess impact. If rate improves, continue. If rate does not improve, return to problem solving.

Decisions What to do if RtI is:

•

Poor

 Was intervention implemented as intended?

• If no - employ strategies in increase implementation integrity • If yes  Is intervention aligned with the verified hypothesis? (Intervention Design)  Are there other hypotheses to consider? (Problem Analysis)  Was the problem identified correctly? (Problem Identification)

Intervention Integrity Decisions

Evidence based intervention linked to verified hypothesis planned Evidence based intervention implemented Student Outcomes (SO) Assessed Treatment Integrity (TI) Assessed Continue Intervention Data-based Decisions Implement strategies to promote treatment integrity Modify/change Intervention From Lisa Hagermoser Sanetti, 2008 NASP Convention

Response to Intervention in Context

Identify the Problem Analyze the Problem Select/Design Intervention Implement Intervention Monitor Progress Evaluate Intervention Effectiveness

L J

Timeline

Progress Monitoring Resources

• • • Interventioncentral.org

Studentprogress.org

Dibels.uoregon.edu

www.interventioncentral.com

www.studentprogress.org

dibels.uoregon.edu

School Level Data Review Worksheet

• • Your project ID is: Last 4 digits of SS# Last 2 digits of year of birth • • Read the case study Answer six questions using data provided

Review

Consensus Infrastructure Implementation

Review

The steps of problem solving as an integrated, fluid, self correcting process.

Step 1 What’s the Problem?

In order to identify a problem, you’ve got to start with three pieces of data • • • Benchmark level of performance Student level of performance Peer level of performance

Step 1 What’s the Problem?

Decision Making Rubric for use with School-Wide Screening

Is this an individual student problem or a larger systemic problem?

Are over 20% of students struggling? Are between 5% and 20% of students struggling? Are 5% or fewer students struggling? Go to problem definition Examine instruction, curriculum, and environment for needed adaptations and develop group intervention Develop small group intervention Go to intervention evaluation adapted from: Heartland AEA 11, Improving Children’s Educational Results

100 90 80 70 60 50 40 30 20 10 0

B A S E L I N E 35 % = Peer Group Benchmark = Aim Line 75 %

Step 2- Why is it occurring?

Goal: The development of hypotheses about probable causes for the identified problem.

Assessments are then conducted to gather information to determine which are most / least likely Prediction statement:

The problem is occurring because ________________.

If ____________ would occur, the problem would be reduced.

DOMAINS

I Instruction C Curriculum E Environment L Learner

Assessment:

How Do We Confirm Hypothesis?

R Review I Interview O Observe T Test

Step 3- What are we going to do about it?

• Effective teaching strategies consider both

what

to teach and

how

to teach it.

• Making good decisions will increase student progress.

• It is critical that the instruction be matched to the problem.

Howell & Nolet, 2000

Step 4- Is it working?

Progress Monitoring Making instructional decisions based on the review and analysis of student data Progress monitoring always includes graphing Goal

Classroom Intervention I Classroom Intervention 2

Intervention

Traditional

Intervention

Traditional vs.

Response to Intervention

J J L

Intervention

L

Consider ESE

J L

Monitor Progress Intervention

J L

Monitor Progress Regular Education

Response to Intervention

Intervention

J

Consider ESE If necessary

Tiers of Service Delivery

Response to Intervention Is it working?

Problem Identification I What is the Problem?

II III Problem Analysis Why is it occurring?

What are we going to do about it?

Intervention Design

What is RtI?

RTI is the practice of (1) providing

high-quality instruction/intervention

matched to student needs and (2) using

learning rate

over time and

level of performance

to (3) make

important educational decisions

to guide instruction. National Association of State Directors of Special Education, 2005

Perceptions of Skills Survey

• • Your project ID is: Last 4 digits of SS# Last 2 digits of year of birth

Final Thoughts

Problem Solving & Response to Intervention

Training Evaluation

No project ID needed

Thanks!

Resources

http://www.oswego.edu/~mcdougal/web_site_4_11_2 005/index.html

(Graphing made easy) http://www.jimwrightonline.com/php/chartdog_2_0/ch artdog.php

(Chart Dog : on-line graphing tool) http://www.studentprogress.org/ http://www.progressmonitoring.org/

Resources

• • • • Armstrong, J., & Anthes, K. (2001). How data can help: Putting information to work to raise student achievement.

American School Board Journal

,

188

, 38-41 Coburn, C., & Talbert, J. (2006). Conceptions of evidence use in school districts: Mapping the terrain.

American Journal of Education

,

112

, 469-495.

Kerr, K., Marsh, J., Ikemoto, G., Darilek, H., & Barney, H. (2006). Strategies to promote data use for instructional improvement: Actions, outcomes, and lessons from three urban districts.

American Journal of Education

,

112

, 496-520.

Young, V. (2006). Teachers’ use of data: Loose coupling, agenda setting, and team norms.

American Journal of Education

,

112

, 521 548.