Transcript Document

Evaluating Research
on Challenging Behavior
CHRIS BORGMEIER
PORTLAND STATE UNIVERSITY
From
Spencer, Detrich & Slocum, 2012
www.pbis.org
Horner, R., & Sugai, G. (2008). Is
school-wide positive behavior support
an evidence-based practice? OSEP
Technical Assistance Center on Positive
Behavioral Interventions and Support.
http://www.pbis.org/files/101007evide
ncebase4pbs.pdf.
Evidence
Basics
From George Sugai
Why evidence-based?
 Maximize outcomes
 Minimize harm
 Increased accountability
 Increase efficiency
 Improve decision making
 Improve resource use
Basic Approach
 Start w/ what has greatest likelihood of
addressing (evidence-based) confirmed
problem/question
 Explained/supported
conceptually/empirically
 Adapt to local context/culture/need
 Monitor regularly & adjust based on data
 Adapt for efficient & durable
implementation
4 Evaluation Criteria
 Effectiveness
 Has/will practice produced desired outcome?
 Efficiency
 What are costs (time, resources, $) to implement
practice?
 Relevance
 Is practice & outcomes appropriate for situation?
 Conceptually soundness
 Is practice based on theory?
Start
Review questions
& data on regular
basis
Does problem
exist?
Yes
Basic Practices
Evaluation
Specify features of
need/problem
No
Identify practice
that addresses
need/problem?
Is practice
research
based?
No
Yes
Yes
Can practice
be adapted?
Yes
No
Implement &
monitor effects
Is adequate
progress
observed?
Yes
Improve efficiency
& sustainability of
practice
implementation
Is evidence of
effectiveness
available?
No
No
Consider another
practice
Start
Review questions
& data on regular
basis
Does problem
exist?
Yes
Specify features of
need/problem
No
Identify practice
that addresses
need/problem?
Identify practice
that addresses
need/problem?
Is practice
research
based?
No
Yes
Yes
Can practice
be adapted?
Yes
Is evidence of
effectiveness
available?
No
No
Consider another
practice
Yes
Can practice
be adapted?
Yes
No
Implement &
monitor effects
Is adequate
progress
observed?
Yes
Improve efficiency
& sustainability of
practice
implementation
Design Questions
 Has functional or cause-effect relationship
been demonstrated & replicated?
 Have alternative explanations been
accounted & controlled for?
 Have threats or weaknesses of methodology
been controlled for?
 Was study implemented w/
fidelity/accuracy?
Research Designs
 Experimental - RCT & SSR
 Evaluation - Descriptive w/ baseline
 Case Study - Descriptive w/o baseline
 Testimonial - No/Limited data
Results Questions
 Who were subjects?
 How much like my participants?
 Where was study conducted?
 How much like where I work?
 What measures were used?
 Do I have similar data?
 What outcomes were achieved?
 Are expected outcomes similar
Effectiveness Logic
 Significance (“believe”)
 Likelihood of same effect by chance
 Effect Size (“strength”)
 Size of effect relative to business as usual
 Consequential Validity (“meaning”)
 Contextually meaningful
PBIS
Evidence
Base
School-wide PBIS – Let’s compare!
 www.pbis.org
 Click on ‘Resource Catalog’
 Then ‘Literature List’
 Evidence Base for SW-PBIS
 Randomized Control Trials
Randomized Control Trials of SW-PBIS
Tier 1/ Universal SW-PBIS

Bradshaw, C. P., Mitchell, M. M., & Leaf, P. J. (2010).Examining the effects of schoolwide positive behavioral interventions and supports on student outcomes: Results from a
randomized controlled effectiveness trial in elementary schools. Journal of Positive
Behavior Interventions, 12(3), 133-148.

Bradshaw, C.,Koth, C., Bevans, K., Ialongo, N., & Leaf, P. (2008). The impact of schoolwide positive behavioral interventions and supports (PBIS) on the organizational health
of elementary schools.School Psychology Quarterly.

Bradshaw, C., Reinke, W., Brown, L., Bevans, K., & Leaf, P. (2008).Implementation of
school-wide positive behavioral interventions and supports (PBIS) in elementary schools:
Observations from a randomized trial. Education and Treatment of Children, 31, 1-26.

Horner, R. H., Sugai, G., Smolkowski, K., Eber, L., Nakasato, J., Todd, A. W., &
Esperanza, J., (2009). A randomized, wait-list controlled effectiveness trial assessing
school-wide positive behavior support in elementary schools. Journal of Positive
Behavior Interventions, 11(3), 133-144.

Sprague, J., & Biglan, A., et al (in progress).A Randomized Control Trial of SWPBS with
Middle Schools.
90-School RCT Study
Horner et al., in press
• Schools that receive technical assistance from typical
support personnel implement SWPBS with fidelity
• Fidelity SWPBS is associated with
▫ Low levels of ODR
▫
▫
Improved perception of safety of the school
▫
▫
.29/100/day v. national mean .34
reduced risk factor
Increased proportion of 3rd graders who meet state reading
standard.
RCT Project Target
Bradshaw & Leaf, in press
 PBIS (21 v. 16) schools reached & sustained high
fidelity
 PBIS increased all aspects of organizational health
 Positive effects/trends for student outcomes
Fewer ODRs (majors + minors)
 Fewer ODRs for truancy
 Fewer suspensions
 Increasing trend in % of students scoring in advanced &
proficient range of state achievement test

Impact Factor
Ask a Faculty member
Collaborative Problem Solving
 Visit the website
 http://www.livesinthebalance.org/
 What do I notice?
 A canoe?
 Advertising products for purchase
 Lots of testimonials
 Little bit of research (10 citations listed under research)

let’s take a closer, evaluative look at the research
Evaluating a Research Study
 Abstract
 Introduction & Literature Review
 Research Questions
 Method & Design
 Subjects & Settings / Measures/ Procedures
 Results
 Discussion & Conclusions
 References
Results Questions
 Who were subjects?
 How much like my participants?
 Where was study conducted?
 How much like where I work?
 What measures were used?
 Do I have similar data?
 What outcomes were achieved?
 Are expected outcomes similar
Collaborative Problem Solving
data from CPS website on 6/18/12










Johnson, M., Ostlund, S., Fransson, G., Landgren, M., Nasic, S., Kadesjo, B., Gillberg, C., and Fernell, E. (2012). AttentionDeficit/Hyperactivity Disorder (ADHD) with Oppositional Defiant Disorder (ODD) in Swedish Children: An Open Study of
Collaborative Problem Solving. Acta Paediactrica, in press.
Ollendick, T. H. (2011). Invited Address: Effective Psychosocial Treatments for Emotional and Behavioral Disorders in
Youth. University of Stockholm, Sweden.. NO studies have been conducted in school
Fraire, M., McWhinney, E., & Ollendick, T. (2011). The effect of comorbidity on treatment outcome in an ODD sample. In T.
settings;
all research
either with
parents
Ollendick (Chair), Comorbidities in children
and adolescents:
Implications foris
evidence-based
treatment.
Symposia
st
presented at the 41 European Association for Behavioral
Cognitive Therapies,
Reykjavik,
Iceland.
or and
in-patient
clinical
settings
Halldorsdottir, T., Austin, K. & Ollendick, T. (2011). Comorbid ADHD in children with ODD or specific phobia: Implications
for evidence-based treatments. In T. Ollendick (Chair), Comorbidities in children and adolescents: Implications for
evidence-based treatment. Symposia presented at the 41st European Association for Behavioral and Cognitive Therapies,
Reykjavik, Iceland.
Epstein, T., & Saltzman-Benaiah, J. (2010). Parenting children with disruptive behaviors: Evaluation of a Collaborative
Problem Solving pilot program. Journal of Clinical Psychology Practice, 27-40.
Martin, A., Krieg, H., Esposito, F., Stubbe, D., & Cardona, L. (2008). Reduction of restraint and seclusion through
Collaborative Problem Solving: A five-year, prospective inpatient study. Psychiatric Services, 59(12), 1406-1412.
Greene, R.W., Ablon, S.A., & Martin, A. (2006). Innovations: Child Psychiatry: Use of Collaborative Problem Solving to
reduce seclusion and restraint in child and adolescent inpatient units. Psychiatric Services, 57(5), 610-616.
Greene, R.W., Ablon, J.S., Monuteaux, M., Goring, J., Henin, A., Raezer, L., Edwards, G., & Markey, J., & Rabbitt, S. (2004).
Effectiveness of Collaborative Problem Solving in affectively dysregulated youth with
oppositional
defiant disorder:
Initial
These
studies
to not
findings. Journal of Consulting and Clinical Psychology, 72, 1157-1164.
evaluate
Greene, R.W., Biederman, J., Zerwas, S., Monuteaux, M., Goring, J., Faraone, S.V. (2002). Psychiatric
comorbidity, family
dysfunction, and social impairment in referred youth with oppositional defiant disorder.
American
Journal
of Psychiatry,
effectiveness of
CPS
159, 1214-1224.
Greene, R. W., Beszterczey, S. K., Katzenstein T., Park, K., & Goring, J. (2002). Are students with ADHD more stressful to
teach? Patterns of teacher stress in an elementary school sample. Journal of Emotional and Behavioral Disorders, 10, 27-37.
Evaluating the Research studies
2004; Greene et
al
2006; Greene
, Ablon,
Martin
2008; Martin
et al
2010 –
Epstein &
Saltzmann
Subjects
Age & #
47 kids w ODD
4-12 yrs. old
3-14 yrs. old
School-age
12 Kids w ODD
Under 12 yrs.
Settings
Outpatient MH
clinic @ hospital
Inpatient Psyc
unit (13 beds)
Inpatient Psyc
unit (15 beds)
Outpatient
clinic
Procedure
Compare CPS w
parent training (PT)
group
Trained unit
staff (pre/post)
Trained unit
staff (pre/post)
Group CPS
parent training
(pre/post)
Outcome
measure
ODDRS
(unpublished rating
scale created by
Greene;
Improvement
ratings (maternal &
therapist)
Restraints &
seclusion
Restraints &
Seclusions
Eyberg Child
Beh. Inv.
Parent Stress
Index
Outcome
Improved slightly
more than PT
Reduced
Reduced
Improvement
pre to post
What does the research tell us?
 So what do we know?
 Based on 4 evaluation studies

All include children ages 12 or less (2008 study does not specify an
age range; simply ‘school age’)
2 are in inpatient psychiatric hospitals
 1 is an outpatient mental health clinic
 1 is a parent training program
 # in school settings = 0

 The research tells us nothing about the efficacy of CPS in
school settings
What does the research tell us?
 Outcome measures
 ODD Rating Scale (unpublished assessment created by the
author) & improvement ratings from parent & therapist


Reductions in restraint & seclusion (Pre/Post)


Similar scores to parent training
Is this due to student behavior change or adult behavior change?
Eyberg CBI & Parent Stress Index (Pre/Post)
 No studies directly measure changes in student
behavior
Concerns
 Only 4 research studies evaluating CPS in 4 years


2 on parent training (1 individual training & 1 group training)
2 in inpatient psyc facilities
 Make sure research you are looking at takes place in
settings that match your application

E.g. school settings v. treatment centers
 2 of 4 studies have been conducted by the author of the
program

Concern if authors are benefiting financially from sale of the
program
Research on CPS in Schools!
 Schaubman, A., Stetson, E., & Plog, A. (2011). Reducing Teacher Stress by
Implementing Collaborative Problem Solving in a School Setting. School Social
Work Journal, 35(2), 72-93.
 Abstract
 Student behavior affects teacher stress levels and the student-teacher relationship.
In this pilot study, teachers were trained in Collaborative Problem Solving
(CPS), a cognitive-behavioral model that explains challenging behavior as the result
of underlying deficits in the areas of flexibility/adaptability, frustration tolerance,
and problem solving. It was hypothesized that teacher stress would be reduced
when teachers' understanding of the underlying causes of student behavior shifted
to a framework of skills development, and they began using a proactive, positive
approach to misbehavior (CPS), with the support of mental health consultation.
Results showed a significant decrease in teacher stress, as measured by self-report.
Further, discipline referrals were significantly reduced. Limitations of the study and
implications for school mental health consultation are also discussed. (Contains 4
figures.)