Transcript Data base Decision-Making: Evaluating the Impact of School

Data-based Decision-making:
Evaluating the Impact of School-wide
Positive Behavior Support
George Sugai, Rob Horner, Anne Todd,
and Teri Lewis-Palmer
University of Oregon
OSEP Funded Technical Assistance Center
www.pbis.org
Purpose
• Examine the extent to which the logic of School-wide
Positive Behavior Support (PBS) fits your real
experience in schools
• Define the outcomes for School-wide PBS
– Is School-wide PBS related to reduction in problem behavior?
– Is School-wide PBS related to improved school safety?
– Is School-wide PBS related to improved academic performance?
• Define tools for measuring School-wide PBS outcomes
• Examine a problem-solving approach for using office
discipline referral (ODR) data for decision-making
• Provide strategies for using data for decision-making and
action planning
To Improve Schools for Children
• Use evidence-based practices
– Always look for data of effectiveness
• Never stop doing what is working
• Implement the smallest change that will result
in the largest improvement
Measure
Compare
Improvement
Designing School-Wide Systems for Student
Success
Academic Systems
Behavioral Systems
Intensive, Individual Interventions
•Individual Students
•Assessment-based
•High Intensity
1-5%
Secondary Group Interventions
•Some students (at-risk)
•High efficiency
•Rapid response
primary Interventions
•All students
•Preventive, proactive
5-10%
80-90%
1-5%
Intensive, Individual Interventions
•Individual Students
•Assessment-based
•Intense, durable procedures
5-10%
Secondary Group Interventions
•Some students (at-risk)
•High efficiency
•Rapid response
80-90%
primary Interventions
•All settings, all students
•Preventive, proactive
SW Positive
Social Competence,
Behavior Academic Achievement, and Safety
Support
OUTCOMES
Supporting
DecisionMaking
Supporting
Staff Behavior
PRACTICES
Supporting
Student Behavior
Improving Decision-Making
From
To
Problem
Problem
Solution
Problemsolving
Information
Solution
Problem-solving Steps
1. Define the problem(s)
– Analyze the data
2. Define the outcomes and data sources for measuring
the outcomes
3. Consider 2-3 options that might work
4. Evaluate each option:
– Is it safe?
– Is it doable?
– Will it work?
5. Choose an option to try
6. Determine the timeframe to evaluate effectiveness
7. Evaluate effectiveness by using the data
– Is it worth continuing?
– Try a different option?
– Re-define the problem?
Key Features of Effective Data
Systems
•
•
•
•
Data are accurate
Data are very easy to collect
Data are used for decision-making
Data are available when decisions need to be
made
• Data collectors must see the information
used for decision-making
Guiding Considerations
•
•
•
•
Use accessible data
Handle data as few times as possible
Build data collection into daily routines
Establish and use data collection as a
conditioned positive reinforcer
• Share data summaries with those who collect
it
Types of Questions
• Initial Assessment Questions
– What type or which program do we need?
– Where should we focus our efforts?
• Ongoing Evaluation Questions
– Is the program working?
– If no,
 Can it be changed?
 Should we end the program?
– If yes,
 Do we need this program anymore?
 What do we need to do to sustain success?
What Data Should be Collected?
• Always start with the questions you want to answer
• Make data that will answer your question:
– Easy, available, reliable
• Balance between reliability and accessibility
– Systems approach
• Consider logistics
– Who? When? Where? How?
• Two levels
– What is readily accessible?
– What requires extra resources?
When Should Data Decisions Be
Made?
• Natural cycles, meeting times
– Weekly, monthly, quarterly, annually
• Level of system addressed
– Individual: daily, weekly
– School-wide: monthly, quarterly
– District/ Region
– State-level
Basic Evaluation Questions
by School or Program
1. What does “it” look like now?
2. How would we know if are successful?
3. Are we satisfied with how “it” looks?
–
YES:

–
Celebrate
NO:


What do we want “it” to look like?
What do we need to do to make “it” look like that?
4. What can we do to keep “it” like that?
Basic School-wide PBS
Evaluation Questions
by School/ District/ Region
Are our efforts making a difference?
1. How many schools have adopted School-wide PBS?
2. Are schools adopting School-wide PBS to criterion?
3. Are schools who are implementing School-wide PBS
perceived as safe?
4. Are teachers delivering instructional lessons with
fidelity as planned?
5. Is School-wide PBS improving student outcomes?
Is School-wide PBS Having a
Positive Influence on School
Culture?
Using Office Discipline Referral Data
Office Discipline Referrals
and The BIG 5!
• Examine office discipline referral rates and
patterns
– Major Problem events
– Minor Problem events
• Ask the BIG 5 questions:
–
–
–
–
–
How often are problem behavior events occurring?
Where are they happening?
What types of problem behaviors?
When are the problems occurring?
Who is contributing?
where
The BIG 5
How
often
what
when
who
Office Discipline Referral Caution
• Data reflects 3 factors:
– Students
– Staff members
– Office personnel
• Data reflects overt rule violators
• Data is useful when implementation is
consistent
– Do staff and administration agree on officemanaged problem behavior verses classroommanaged behavior?
Staff Managed (minors)
•Tardy
•Unprepared; no
homework/materials
•Violation of classroom
expectations
•Inappropriate language
•Classroom disruption
•Minor safety violation
•Lying/Cheating
Consequences are determined
by staff
Office Managed (majors)
•Repeated minor behaviors
•Insubordination
•Blatant disrespect
•Abusive/Inappropriate language
•Harassment/Intimidation
•Fighting/Physical aggression
•Safety violations that are
potentially harmful to self, others
and/or property
•Vandalism/Property destruction
•Plagiarism
•Theft
•Skipping classes
•Illegal behaviors:
Arson
Weapons
Tobacco
Alcohol/Drugs
General Procedure for Dealing with Problem Behaviors
Observe problem
behavior
Find a place to talk with
student(s)
Is behavior
major?
NO
Determine
consequence
Problem solve
Determine
consequence
Follow procedure
documented
File necessary
documentation
Ensure safety
Write referral and
Escort student to office
Problem solve
NO
YES
Does
student
have 3?
Follow
documented
procedure
YES
Follow
through with
consequences
Send
referral to
office
File necessary
documentation
Follow up
with student
within a
week
Office Discipline Referral Form
Name: _________________________ Grade: _____ Date: _____
Referring Person: ________________________Time: ________
Others involved: None Peers Staff Teacher Substitute Unknown Other
Problem Behavior
Major
Abusive
language
Fighting/
physical
aggression
Harassment
Overt
defiance
Other_____
Minor
Inappropriate
language
Disruption
Property
misuse
Noncompliance
Other_____
Consequence
Lose privilege
individual instruction
Conference
In-school suspension
Location
Hallway
Cafeteria
Library
Restroom
Office
Parking lot
Classroom
On bus
Special
event
Common
area
Other_____
Possible Motivation
Attention
from peers
Attention
from adults
Avoid peers
Avoid adults
Parent contact
Out-of-school suspension
Time in office
Other________________
Avoid work
Obtain items
Don’t know
Other_____
SWIS™ Compatibility Checklist
Procedure for Documenting Office Discipline Referrals
School ___________________________
Date ____________________
Date
Date
Compatibility Question
1. Does a clear distinction exist between problem behaviors that are staff
managed versus office managed exist and is it available for staff reference?
Yes
No
Yes
No
2. Does a form exist that is SWIS™ compatible for SWIS™ data entry that
includes the following categories?
Yes
No
Yes
No
a. Student name?
Yes
No
Yes
No
b. Date?
Yes
No
Yes
No
c. Time of incident?
Yes
No
Yes
No
d. Student’s grade level?
Yes
No
Yes
No
e. Referring staff member?
Yes
No
Yes
No
f. Location of incident?
Yes
No
Yes
No
g. Problem behavior?
Yes
No
Yes
No
h. Possible motivation?
Yes
No
Yes
No
i. Others involved?
Yes
No
Yes
No
j. Administrative decision?
Yes
No
Yes
No
k. Other comments?
Yes
No
Yes
No
l. No more than 3 extra info.
Yes
No
Yes
No
3. Does a set of definitions exist that clearly defines all categories on the office
discipline referral form?
Yes
No
Yes
No
Next review date: _______________
Redesign your form until answers to all questions are “Yes.”
Readiness requirements 4 and 5 are complete when you have all “Yes” responses.
Tables versus Graphs
2001
Aug
Number of
Days
0
2001
Sep
19
5
0.26
2001
Oct
21
18
0.86
2001
Nov
18
17
0.94
2001
Dec
14
21
1.50
2002
Jan
22
18
0.82
2002
Feb
17
15
0.88
2002
Mar
19
26
1.37
2002
Apr
21
14
0.67
2002
May
18
13
0.72
2002
Jun
11
2
0.18
2002
Jul
0
0
0.00
180
149
0.83
Year
Totals:
Month
Number of
Referrals
0
Average Referrals
Per Day
0.00
Number of ODR per Day and
Month
Total verses Rate
Total
Ref
versus
Ref/Day/Mo
Total Number of ODRs per Month
NV High School
70
60
Referrals
50
40
30
20
10
0
Aug Sept Oct
Nov Dec
Jan
Feb
School Month
Mar
Apr
May Jun
Number
ODRs
per Ref/Day/Mo
Day and Month
TotalofRef
versus
Mean Referrals per Day
5
4
3
2
1
0
Aug Sept Oct
Nov
Dec
Jan
Feb
School Month
Mar
Apr
May
Jun
Priorities and Rationale
• Graphs
• Rate
SWIS summary 2008-2009 (Majors Only)
3,410 schools; 1,737,432 students; 1,500,770 ODRs
Grade Number of
Range
Schools
Avg. Enrollment
per school
National Avg. for Major ODRs per
100 students, per school day
K-6
2,162
450 .34 = about 1 Major ODR every 3
school days, or about 34 every 100
days
6-9
602
657 .85 = a little less than 1 Major ODR
per school day, or about 85 every 100
days
9-12
215
887 1.27 = more than 1 Major ODR per
school day, or about 127 every 100
days
K(8-12)
431
408 1.06 = about 1 Major ODR per school
day, or about 106 every 100 days
Newton, J.S., Todd, A.W., Algozzine, K, Horner, R.H. & Algozzine, B. (2009). The Team Initiated
Problem Solving (TIPS) Training Manual. Educational and Community Supports, University of Oregon
SWIS summary 2009-10 (Majors Only)
4,019 schools; 2,063,408 students; 1,622,229 ODRs
Grade Range
Number of
Schools
Mean Enrollment Median ODRs
per school
per 100 per
school day
K-6
2565
452
.22
6-9
713
648
.50
9-12
266
897
.68
K-(8-12)
474
423
.42
Interpreting Office Referral Data:
Is there a problem?
• Absolute level (depending on size of
school)
– Middle, High Schools (> 1 per day per 100)
– Elementary Schools (>1 per day per 300)
• Trends
– Peaks before breaks?
– Gradual increasing trend across year?
• Compare levels to last year
– Improvement?
Application Activity: Absolute Value
Is there a Problem?
Compare with national average:
# per dayReferrals
per 100 students
per School Day
Office Discipline
5.75
625/100 = 6.25
6.25 X .92 =
Middle School of 625 students?
Office Discipline Referrals per Day per Month per 100 Students
16
14
12
10
8
Series1
6
4
2
0
Sept
Oct
Nov
Dec
Jan
Feb
March
April
May
June
Office Re fe rrals pe r Day pe r M onth
A v e R efer r als per D ay
Compare with National Average Last150
/ 100 = 1.50
year
Elementary School with 150 Students
1.50 X .35 = .53
2
1.5
1
0.5
0
Sept
Oct
Nov
Dec
Jan
Feb
Mar
School Months
Apr
May
Jun
Office
Re
fe
rrals
pe
r
Day
pe
r
M
onth
Compare with National Average
1800 / 100 = 18
18 X 1.06 =
Last year
A v e R efer r als per D ay
19.08
20
High School of 1800 students
15
10
5
0
Sept
Oct
Nov
Dec
Jan
Feb
Mar
School Months
Apr
May
Jun
Office Re fe rrals pe r Day pe r M onth
A v e R efer r als per D ay
This Year
20
Middle School of 700 students
15
10
5
0
Sept
Oct
Nov
Dec
Jan
Feb
Mar
School Months
Apr
May
Jun
Office Re fe rrals pe r Day pe r M onth
A v e R efer r als per D ay
Last Year and This Year
20
15
10
5
0
Sept
Oct
Nov
Dec
Jan
Feb
Mar
School Months
Apr
May
Jun
Office Re fe rrals pe r Day pe r M onth
A v e R efer r als per D ay
Last Year and This Year
20
15
10
5
0
Sept
Oct
Nov
Dec
Jan
Feb
Mar
School Months
Apr
May
Jun
Middle
School
Office Re fe rrals pe r Day pe r M onth
This Year
A v e R efer r als per D ay
N= 495
20
15
10
5
0
Sept
Oct
Nov
Dec
Jan
Feb
School Months
Mar
Apr
May
N= 495
Is There a Problem? #2
Absolute - Trend - Compare
Office Re fe rrals pe r Day pe r M onth
T his year (Middle)
A v e R efer r als per D ay
Middle
School
20
15
10
5
0
Sept
Oct
Nov
Dec
Jan
Feb
Mar
School Months
Apr
May
Jun
Middle
School
N= 495
Office Re fe rrals pe r Day pe r M onth
A v e R efer r als per D ay
Last Year and This Year
20
15
10
5
0
Sept
Oct
Nov
Dec
Jan
Feb
Mar
School Months
Apr
May
Jun
Middle
School
N= 495
Office Re fe rrals pe r Day pe r M onth
A v e R efer r als per D ay
Last Year and This Year
20
15
10
5
0
Sept
Oct
Nov
Dec
Jan
Feb
Mar
School Months
Apr
May
Jun
Trevor Test Middle School
565 students
Grades 6,7, and 8
Office Referrals per Day per Month
Referrals per Prob Behavior
This Year
50
15
10
Lang.
5
0
Defiance
Sept
Oct
Nov
Dec
Jan
Feb
Mar
Apr
May
Jun
Number of Referrals
Ave Referrals per Day
20
40
30
20
10
Disrespect
School Months
0
Lang
Achol
Arson
Bomb Combs Defian Disrupt Dress Agg/fgt Theft Harass Prop D
Skip
Tardy Tobac
Vand
Weap
Types of Problem Behavior
Referrals by Location
Referrals by Time of Day
60
40
20
Bath R Bus A
Bus
Caf
Class
Comm
Gym
Hall
Libr
Play G
Spec
Other
School Locations
Cafeteria Class Commons
12:00
40
30
20
10
0
Students per Number of Referrals
7:00
7:30
8:00
8:30
9:00
9:30
10:00 10:30 11:00 11:30 12:00 12:30
1:00
1:30
2:00
2:30
3:00
3:30
Time of Day
Hall
100
Number of Students
0
Harrass Skip
50
Number of Referrals
Number of Office Referrals
80
80
60
40
20
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
Number of Referrals
Langley Elementary School
478 Students
Kindergarten - Grade 5
Total Office Discipline Referrals
FRMS Total Office Discipline Referrals
3000
2500
2000
1500
1000
500
0
94-95 95-96 96-97 97-98 98-99 99-00 00-01 01-02
Academic Years
What Does a Reduction of 850
Office Discipline Referrals
and 25 Suspensions Mean?
Kennedy Middle School
• Savings in Administrative
Time
• ODR = 15 minutes/ event
• Suspension = 45
minutes/event
• 13,875 minutes
• 231 hours
• 29, 8-hour days
• Savings in Student
Instructional Time
• ODR = 45 minutes/ event
• Suspension = 216
minutes/ event
• 43,650 minutes
• 728 hours
• 121, 6-hour school days
Is Implementation Related to
Reduction in Problem Behavior?
767 students
Mean Referrals Per School Day
FRMS Discipline Referrals
1994-95; 1995-96
25
20
15
10
5
0
Sept Oct Nov Dec Jan Feb Mar Apr May June
School Months
Are Schools Adopting School-wide
PBS to Criterion?
• Use the:
– Team Implementation Checklist (TIC)
– School-wide Evaluation Tool (SET)
– EBS Self-Assessment Survey (School-wide section)
– Other
• Measure and analyze annually
We’ll focus on TIC today!
Team Implementation Checklist (TIC)
• Characterizes the evolution of School-wide
PBS implementation:
– “Achieved,” “In progress,” or “Not started”
• Assists in:
– Initial assessment
– Getting started on action plan
– Measuring progress of School-wide PBS
Implementation
• Assesses team-based response
– Quarterly or monthly
TIC Feature Areas
1. Establish Commitment
2. Establish and Maintain Team
3. Conduct Self-Assessment
4. Define Expectations
5. Teach Expectations
6. Establish Reward System
7. Establish Violations System
8. Establish Information System
9. Build Capacity for Function-based Support
10.Ongoing Activities
Team Implementation Checklist
Checklist #1: Start-Up Activity
Complete and submit Monthly.
Status: Achieved, In Progress, Not Started
Date:
(MM/DD/YY)
Establish Commitment
1. Administrator’s support and active
involvement.
Status
:
2. Faculty/Staff support (One of top 3 goals,
80% of faculty document support, 3 year
timeline).
Status
:
Establish and Maintain Team
3. Team established (representative).
Status
:
4. Team has regular meeting schedule,
effective operating procedures.
Status
:
5. Audit is completed for efficient integration of
team with other teams/initiatives addressing
behavior support.
Status
:
Self-Assessment
6. Team/faculty completes EBS selfassessment survey.
Status
:
7. Team summarizes existing school discipline
data.
Status
:
Team Implementation Checklist continued
8. Strengths, areas of immediate focus and
action plan are identified.
Status
:
Establish School-wide Expectations
9. 3-5 school-wide behavior expectations are
defined.
Status
:
10. School-wide teaching matrix developed.
Status
:
11. Teaching plans for school-wide
expectations are developed.
Status
:
12. School-wide behavioral expectations taught
directly and formally.
Status
:
13. System in place to acknowledge/reward
school-wide expectations.
Status
:
14. Clearly defined and consistent
consequences and procedures for undesirable
behaviors are developed.
Status
:
Establish Information System
15. Discipline data are gathered, summarized,
and reported.
Status
:
Build Capacity for Function-based Support
16. Personnel with behavioral expertise are
identified and involved.
Status
:
17. Plan developed to identify and establish
systems for teacher support, functional
assessment and support plan development and
implementation.
Status
:
Team Implementation Checklist continued
Checklist #2: On-going Activity Monitoring
Complete and submit Monthly.
Status: Achieved, In Progress, Not Started
1. EBS team has met at least monthly.
Status
:
2. EBS team has given status report to faculty
at least monthly.
Status
:
3. Activities for EBS action plan implemented.
Status
:
4. Accuracy of implementation of EBS action
plan assessed.
Status
:
5. Effectiveness of EBS action plan
implementation assessed.
Status
:
6. EBS data analyzed.
Status
:
Scoring the TIC
• Implementation Points
– Achieved = 2
– In progress = 1
– Not Started = 0
• Percentage of Items Implemented
– Total
 Number of items scored as “Achieved” divided by 17 (items)
– Subscale scores
 Number of items in each subscale area scored as “Achieved”
divided by the number of items in that subscale area
• Percentage of Points Implemented
– Total
 Total number of points divided by 34
– Subscale scores
 Total number of points in each subscale divided by total number of
items multiplied by 2
Team Implementation Checklist (TIC)
Team Implementation Checklist (TIC)
Total Average TIC Scores
School
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Nov-03 Dec-03 Apr-04 Oct-03 Dec-03 Mar-04 Sep-03 Oct-03 Nov-03 Feb-04 Mar-04 May- Mar-03 Nov-03 Jan-04 Apr-04
04
1
2
3
Schools
4
100
90
80
70
60
50
40
30
20
10
0
ov
er
al
l
Ex
pe
ct
at
io
In
ns
fo
rm
at
io
n
Fu
Sy
nc
st
tio
em
nba
se
d
Su
pp
or
t
en
t
SW
As
se
ss
m
Se
lf
Te
am
it m
en
t
Pre-PBS
Co
m
m
% implemented
Team Implementation Checklist
1. What is working well? 2. What are next steps?
1. What is going well?
ov
er
al
l
Ex
pe
ct
at
io
In
ns
fo
rm
at
io
n
Fu
Sy
nc
st
em
tio
nba
se
d
Su
pp
or
t
en
t
SW
As
se
ss
m
Se
lf
Te
am
it m
en
t
100
90
80
70
60
50
40
30
20
10
0
Co
m
m
% implemented
Team Implementation Checklist
Pre-PBS
2. What are next steps?
Year 1
ov
er
al
l
Ex
pe
ct
at
io
In
ns
fo
rm
at
io
n
Fu
Sy
nc
st
em
tio
nba
se
d
Su
pp
or
t
en
t
SW
As
se
ss
m
Se
lf
Te
am
it m
en
t
100
90
80
70
60
50
40
30
20
10
0
Co
m
m
% implemented
Team Implementation Checklist
Pre-PBS
1. What is going well?
Year 1
2. What are next steps?
Year 2
Main Messages
• Invest in prevention
• Create an effective environment
– Leadership, teams; hosts for effective practices
• Use different systems for different problems
– Individual student level alone will be insufficient
– Collaboration with Mental Health Professionals
• Build a culture of competence
– Define, teach, monitor, and reward appropriate behavior
• Build sustainable systems
– Resist person-dependent interventions
• Invest in gathering and using information for decisionmaking and problem-solving
Action Planning
• Use your self-assessment information
– Rally School-wide commitment
– Establish a PBS Team
– Focus on prevention (define, teach, monitor, and
reward appropriate behavior)
 Ask kids tomorrow if they know the expectations
 Ask kids if they are being acknowledged for appropriate
behavior
– Use information system to guide implementation
efforts
• Build Action Plan
– When will the team meet?
– What will be reported to faculty?
– What will be reported to families?
Action Planning
• Which system are you going to work on?
• What are the specific outcomes?
– When will they be completed?
– What short-term activities are needed?
– Who will be responsible?
• Reporting Schedule
– What information will be gathered and by whom?
– When will information be reported?