Transcript Using Data for Decision-making Rob Horner, Anne Todd, Steve Newton, Bob Algozzine, Kate Algozzine.

Using Data for Decision-making
Rob Horner, Anne Todd, Steve Newton,
Bob Algozzine, Kate Algozzine
Main Ideas

Decisions are more likely to be effective and
efficient when they are based on data.

The quality of decision-making depends most
on the first step (defining the problem to be
solved)

Define problems with precision and clarity
Main Ideas

Data help us ask the right questions…they do
not provide the answers: Use data to




Identify problems
Refine problems
Define the questions that lead to solutions
Data help place the “problem” in the context
rather than in the students.
Main Idea

The process a team uses to problem solve is
important:

Roles:


Facilitator; Recorder; Data analyst; Active member
Organization




Agenda; Old business (did we do what we said we would
do); New business; Action plan for decisions.
What happens BEFORE a meeting
What happens DURING a meeting
What happen AFTER a meeting
Main Ideas


Build “decision systems” not “data systems”
Use data in “decision layers”


Is there a problem? (overall rate of ODR)
Localize the problem





(location, problem behavior, students, time of day)
Get specific
Don’t drown in the data
It’s “OK” to be doing well
Be efficient
?
Beh
Proble m
Location
?
Time
of
Day
Student
Setting
A
B
C
D
E
F
G
H
I
J
Loc ations
1
2
3
4
5
6
7
8
9 10
Tim es
A
B
C
D
E
F
G
H
I
Students
1
2
3
4
5
6
7
8
J
9 10
K
K
11
Using Data


Do we have a problem?
Refine the description of the problem?


Test hypotheses




What behavior, Who, Where, When, Why
“I think the problem on the playground is due to Eric”
“ We think the lunch period is too long”
“We believe the end of ‘block schedule” is used
poorly”
Define how to monitor if solution is effective
Review
Status and
Identify
Problems
Team Initiated
Problem Solving
(TIPS) Model
Evaluate and
Revise
Action Plan
Develop and
Refine
Hypotheses
Collect
and Use
Data
Discuss and
Select
Solutions
Develop and
Implement
Action Plan
Problem Solving
Foundations
Identifying problems/issues

What data to monitor




What question to answer


Do we have a problem?
What questions to ask of Level, Trend, Peaks




ODR per day per month
OSS, ISS, Attendance, Teacher report
Team Checklist/ SET (are we doing what we planned to do?)
How do our data compare with last year?
How do our data compare with national/regional norms?
How do our data compare with our preferred/expected status?
If a problem is identified, then ask

What are the data we need to make a good decision?
Total Office Discipline Referrals
Total Office Discipline Referrals as of January 10
Change
Report Options
3.49
2.75
2.5
2.7
1.8
1.4
0
.00
Using Data to Refine Problem
Statement

The statement of a problem is important for teambased problem solving.


Everyone must be working on the same problem with the same
assumptions.
Problems often are framed in a “Primary” form, that
creates concern, but that is not useful for problemsolving.


Frame primary problems based on initial review of data
Use more detailed review of data to build “Solvable Problem
Statements.”
Solvable Problem Statements
(What are the data we need for a decision?)

Solvable problem statements include
information about the five core “W”
questions.





What is problem, and how often is it happening
Where is it happening
Who is engaged in the behavior
When the problem is most likely
Why the problem is sustaining
Primary versus Precision Statements

Primary Statements





Too many referrals
September has more
suspensions than last
year
Gang behavior is
increasing
The cafeteria is out of
control
Student disrespect is
out of control

Precision Statements

There are more ODRs
for aggression on the
playground than last
year. These are most
likely to occur during
first recess, with a large
number of students, and
the aggression is related
to getting access to the
new playground
equipment.
Primary versus Precision Statements

Primary Statements





Too many referrals
September has more
suspensions than last
year
Gang behavior is
increasing
The cafeteria is out of
control
Student disrespect is
out of control

Precision Statements

There are more ODRs
for aggression on the
playground than last
year. These are most
likely to occur during
first recess, with a
large number of
students, and the
aggression is related to
getting access to the
new playground
equipment.
Precise or Primary Statement?

Children are using inappropriate language
with a high frequency in the presence of both
adults and other children. This is creating a
sense of disrespect and incivility in the school

James D. is hitting others in the cafeteria
during lunch, and his hitting is maintained by
peer attention.
Precise or Primary Statement?

ODRs during December are higher than in any other
month.

Minor disrespect and disruption are increasing over
time, and are most likely during the last 15 minutes
of our block periods when students are engaged in
independent seat work. This pattern is most
common in 7th and 8th grades, involves many
students, and appears to be maintained by escape
from work (but may also be maintained by peer
attention… we are not sure).
Precise or Primary Statement?

Three 5th grade boys are name calling and
touching girls inappropriately during recess in
an apparent attempt to obtain attention and
possibly unsophisticated sexual expression.

Boys are engaging in sexual harassment
Organizing Data for Decision-making


Compare data across time
Moving from counts to count/month
SWIS summary 2008-2009 (Majors Only)
3,410 schools; 1,737,432 students; 1,500,770 ODRs
Grade Number of
Range
Schools
Avg. Enrollment
per school
National Avg. for Major ODRs per
100 students, per school day
K-6
2,162
450 .34 = about 1 Major ODR every 3
school days, or about 34 every 100
days
6-9
602
657 .85 = a little less than 1 Major ODR
per school day, or about 85 every 100
days
9-12
215
887 1.27 = more than 1 Major ODR per
school day, or about 127 every 100
days
K(8-12)
431
408 1.06 = about 1 Major ODR per school
day, or about 106 every 100 days
Newton, J.S., Todd, A.W., Algozzine, K, Horner, R.H. & Algozzine, B. (2009). The Team Initiated Problem Solving (TIPS) Training Manual.
Educational and Community Supports, University of Oregon unpublished training manual.
Start with the ODR/Day/Month Graph

Use the information in the data to build a
narrative that draws the team into problem
solving.

Be descriptive
Link local data to national patterns
Tie the data back to local conditions/events.


Elementary School
465 students (465/ 100 = 4.6 X .34= 1.56)
Our rate of problem
behavior has been
above the national
average for schools
our size for 9 of 10
months this year.
There has been a
decreasing trend since
Dec.
Elementary School
1000 Students (1000/100 =10 X .34= 3.4)
The rate of problem
behavior has been at
or below the national
average schools our
size for 6 of 10
months. The past 4
months have been
below the national
average
Middle School 765 students (765/100 = 7.6 X .85= 6.46)
The rate of problem
behavior has been at
or below the national
average schools our
size for 9 of 10
months. The past 8
months have been
below the national
average with a
decreasing trend
Describe the narrative for this school
Describe the narrative for this school
Describe the narrative for this school
Describe the narrative for this school
Using Data to Build Precision

Given that we know we have a problem

What problem behaviors
Where are they occurring
When are they occurring
Who is involved
Why do they keep happening




What questions to ask about the
patterns of problem behaviors?
1.
2.
3.
4.
Do we have one problem behavior situations
or more than one?
Do we have many problem behaviors or just
a few problem behaviors?
Do we have clusters of problem behaviors?
What school wide expectations do we need
to re-teach?
Data should allow asking the right question…
not supplying the answer.

If many referrals in class




Which classes?
Which students?
When?
If many referrals in cafeteria



Which students?
What times? (beginning or end of lunch period?)
What problem behaviors?
Disrespect is our most
frequent problem behavior.
We also have incidents of
fighting and harassment
What are next questions?
Who, When, Why?
Our most frequent
problem behavior is
disrespect, followed by
inappropriate language,
disruption and tardy
What are next questions?
Who, When, Why?
We have many instances of
disrespect,
aggression/fighting.
technology violations, tardy,
harassment, lying, skipping,
and inappropriate language
What questions to ask about Referrals
by Location

Where are the problems occurring?

Are there problems in many locations,
clusters of locations, or one location?
Many problem behaviors in
class
Many problem behaviors in
unstructured settings (hall,
playground, parking lot,
bathroom)
Many problems in the
cafeteria, hallway, common
area, class, bathroom.
Where is the ‘unknown’
location?
What questions to ask about Referrals
by Time

When are the problem behaviors occurring?

How do those times match with the daily
activities?

How does this information match up to
Referrals by Location?
Most problems are
occurring between 9:4510:45.
Other problematic times
are 8-8:45 and 11:30.
Most problems
are occurring at
noon
Many problems at 12:15, 7:45-8:30,
10:00-10:45
What questions to ask about Referrals
by Student

What proportion of students has 0-1 ODR?

What proportion of students has 2-5 ODRs?

What proportion of students has 6+ ODRs?

Do we have systems of support that increase
student success?
Student # 121 needs
individualized support.
8 students are likely candidates
for some type of Tier II support.
87% of our students have received
0-1 ODR
14 students are likely candidates
for some type of Tier II support.
Student #119 needs individualized
support
We have 11 students who are
likely candidates for some
type of Tier II support
93% of our students have
received no more than one
ODR
What questions to ask about Referrals
by Perceived Motivation

What is perceived as maintaining the problem
behavior?

Are there one or more perceptions?
The problem behaviors are most likely
maintained by task avoidance and peer
avoidance.
We have many incidents with unknown
motivation
Problem behaviors appear to be
maintained by peer and adult attention
Problem behaviors
appear to be
maintained by
escaping adult
attention
Using the TIPS Minutes to guide
Problem Solving
1.
2.
Write the precision problem statement in the
left hand column below
Define a goal and write it in the right hand
column
Using Data to Build Solutions

Prevention: How can we avoid the problem context?



Who, When, Where
Schedule change, curriculum change, etc
Teaching: How can we define, teach, and monitor what we want?


Teach appropriate behavior
Use problem behavior as negative example

Recognition: How can we build in systematic reward for desired behavior?

Extinction: How can we prevent problem behavior from being rewarded?

Consequences: What are efficient, consistent consequences for problem
behavior?

How will we collect and use data to evaluate (a) implementation fidelity, and
(b) impact on student outcomes?
Solution Development
Prevention
Teaching
Reward
Extinction
Corrective Consequence
Data Collection
Trevor Test Middle School
565 students
Grades 6,7,8
Trevor Test Middle School
Is there a problem? If so, what is it?
Avg. ODRs Per School Day
20
15
10
5
0
Sep
Oct
Nov
Dec
Jan
School Months
School Avg.
National Avg. = 4.8
140
120
100
80
60
40
Number of Referrals
Referrals by Location
200
180
20
0
Student No.
5:00 PM
4:30 PM
4:00 PM
3:30 PM
3:00 PM
2:30 PM
2:00 PM
1:30 PM
1:00 PM
12:30 PM
12:00 PM
11:30 AM
11:00 AM
Referrals by Problem Behavior
10:30 AM
10:00 AM
9:30 AM
9:00 AM
8:30 AM
40
30
20
8:00 AM
60
50
7:30 AM
90
80
70
Number of Referrals
120
110
100
7:00 AM
Number of Referrals
140
130
1
13
16
18
2
20
24
28
30
33
38
4
9
17
21
37
43
23
31
39
40
41
5
8
11
29
12
22
25
35
42
6
14
34
15
26
36
7
3
19
32
27
10
Café
Hall
Common
Class
Other
Special evt
Bus
Bus Zn
Gym
Bathrm
Library
Music rm
Stadium
Off-Campus
Locker rm
Office
Unknown
Park lot
Minor
Tardy
Bomb
Arson
Weapons
Other
Unknown
Drugs
M-Prpty Misuse
M-Other
M-Dress
M-Tech
Tech
Inapp affection
Out bounds
M-Unknown
Gang display
Skip
Truan
Lying
M-Disruption
Dress
Tobacco
Alcohol
Combust
M-Inapp lan
Forge/Theft
Vandal
M-Contact
M-Disrespt
Prop dam
Agg/Fight
M-Tardy
Skip
Harass
Disrespt
Inapp lan
Disruption
10
0
Plygd
Number of Referrals
Trevor Test Middle School
11/01/2007 through 01/31/2008 (last 3 mos.)
Referrals by Time
130
120
110
100
90
80
70
60
50
40
30
20
10
0
Referrals by Student
80
160
70
60
50
40
30
20
10
0
Precise Problem Statement &
Hypothesis Development

Many students from all grade levels are engaging in
disruption, inappropriate language and harassment in
cafeteria and hallway during lunch, and the behavior
is maintained by peer attention

A smaller number of students engage in skipping
and noncompliance/defiance in classes, (mostly in
rooms 13, 14 and 18), and these behaviors appear to
be maintained by escape.
Solution Development
Prevention
Teaching
Reward
Extinction
Corrective Consequence
Data Collection
Solution Development:
For disruption in hall and cafeteria
Prevention
Teaching
*Teach behavioral expectations in
cafeteria
*Maintain current lunch schedule,
but shift classes to balance numbers.
Reward
Establish “Friday Five”: Extra 5 min
of lunch on Friday for five good
days.
Extinction
Encourage all students to work for
“Friday Five”… make reward for
problem behavior less likely
Corrective Consequence
Active supervision, and continued
early consequence (ODR)
Data Collection
Maintain ODR record and
supervisor weekly report
Phoenix Elementary
Using Data For Decision-Making
You are the Behavior Support team for Phoenix
Elementary. 265 students k-5





Do you have a problem?
Where?
With Whom?
What other information might you want?
Given what you know, what considerations
would you have for possible action?
Examples

Phoenix Elementary

What is national comparison?





265/100 = 2.65 2.65 X .34 = .90
Absolute level compared with last year, compared
with teacher/staff impressions, compared with
family impressions, compared with student
impressions.
Where, what, when, who , why
Hypotheses?
Solutions
M ean Student C ontac ts per D a
Phoenix
Student DisciplineContacts
5
4
3
Year 2
Year 1
2
1
0
Sept
Oct
Nov
Dec
Jan
Feb
Mar
School M onths
Apr
May
June
N um ber of Student C ontac ts
Phoenix Elementary
Locations
140
120
100
80
60
Year 1
Year 2
40
20
0
Playgd Class Restrm Caf
Location
Other
Phoenix Elementary ODR per Student
Major ODRs Year 2 Only
14
12
10
8
6
4
2
Students
49
46
43
40
37
34
31
28
25
22
19
16
13
10
7
4
0
1
Number of Student contacts
16
7:
00
7:
30
8:
00
8:
30
9:
00
9:
30
10
:00
10
:30
11
:00
11
:45
12
:15
12
:45
1:
15
1:
45
2:
15
2:
45
3:
15
Number of Referrals
Phoenix Elementary ODR per Time of Day
30
25
20
15
10
5
0
Time of Day
Problem Statement


Do we have a problem?
Build a precise problem statement
Solution Development
Prevention
Teaching
Reward
Extinction
Corrective Consequence
Data Collection
Langley Elementary
School
478 Students
K-5
Precision Statement/Hypothesis

What
Where
When
Who
Why
What other info needed?

Possible Solutions?





Solution Development
Prevention
Teaching
Reward
Extinction
Corrective Consequence
Data Collection
Sandhill High school
354 students
Sandhill High School
Is there a problem? If so, what is it?
Avg. ODRs Per School Day
5
4
3
2
1
0
Sep
Oct
Nov
Dec
Jan
Feb
Mar
Apr
School Months
Previous Year
This Year
National Avg. = 3.72
May
Jun
0
50
40
30
20
Number of Referrals
Referrals by Location
80
10
0
5:00 PM
4:30 PM
4:00 PM
3:30 PM
3:00 PM
2:30 PM
2:00 PM
1:30 PM
1:00 PM
12:30 PM
12:00 PM
11:30 AM
10
5
11:00 AM
Referrals by Problem Behavior
10:30 AM
15
10:00 AM
20
9:30 AM
30
25
9:00 AM
25
8:30 AM
40
8:00 AM
30
7:30 AM
50
45
Number of Referrals
35
7:00 AM
Number of Referrals
60
3
4
5
6
7
9
11
12
14
15
16
17
18
19
20
21
24
25
27
28
30
31
32
33
34
35
37
38
39
43
44
45
46
48
49
51
52
1
50
23
8
40
41
2
26
29
36
10
47
53
42
22
13
Class
Unknown
Other
Hall
Café
Library
Park lot
Office
Common
Bathrm
Gym
Stadium
Special evt
Plygd
Off-Campus
Locker rm
Bus Zn
Arson
Bomb
Combust
Dress
Gang display
Iapp affection
M-Contact
M-Disrespt
M-Disruption
M-Dress
M-Inapp lan
M-Other
M-Prpty Misuse
M-Tardy
M-Tech
M-Unknown
Other behav
Out bounds
Tech
Tobacco
Unknown behav
Vandal
Alcohol
Weapons
M-Warning
Prop dam
Theft
Tardy
Lying
Drugs
Inapp lan
Harass
Agg/Fight
Disruption
Disrespt
Skip
0
Bus
Number of Referrals
Sandhill High School
02/01/2007 through 04/30/2007 (last 3 mos.)
Referrals by Time
55
35
20
15
10
5
Referrals by Student
14
70
60
12
10
8
6
4
2
0
Precision Statement/Hypothesis

What
Where
When
Who
Why
What other info needed?

Possible Solutions?





Solution Development
Prevention
Teaching
Reward
Extinction
Corrective Consequence
Data Collection
Summary





Establish information systems for decisionmaking… not data systems for reporting.
Use data to identify problems
Use data to clarify/refine problems
Use knowledge of context and content to
build solutions.
Use data to monitor impact of solutions.