Transcript Identifying Effective Interventions & Practices

Ways of Knowing & Research
Based Practices
CHRIS BORGMEIER, PHD
PORTLAND STATE UNIVERSITY
Group Activity
 Review Readings
 Stanovich Ch. 2
 Stanovich & Stanovich
“We associate truth with convenience, with
what most closely accords with self-interest
and personal well-being, or promises best to
avoid awkward effort or unwelcome
dislocation of life. We also find highly
acceptable what contributes most to selfesteem.”
John Kenneth Galbraith

“Conventional wisdom must be simple,
convenient, comfortable, and comforting…
not necessarily true.”
Steven Levitt
 “Social behavior is complex, and to
comprehend its character is mentally tiring.
Therefore we adhere, as though to a raft, to
those ideas which represent our
understanding”
John Kenneth Galbraith
Ways of Knowing
 Personal experience

Research can stimulate, inform, reinforce, challenge & question our
own experiences to enhance professional judgment
 Tradition

Simply accept what has been done as the bet or right way (eliminates
the need to search for knowledge & understanding)
 Authority

People considered to experts or authorities are major sources of
knowledge
 Challenge = these ‘ways of knowing’ are primarily
idiosyncratic, informal & influenced heavily by subjective
interpretation
Ways of Knowing
 Research
 Involves a systematic process of gathering, interpreting and
reporting information
 Disciplined inquiry characterized by accepted principles to
verify that claim is reasonable
Types of Research
 Basic Research – formulates & refines theories
 Applied Research – improves practice & solves
practical problems
 Action Research – goal is to solve a specific
classroom or school problem, improve practice or
help make a decision at as single site
What to look for in articles
Refereed v. Non-refereed articles
 Refereed articles – reviewed by panel of peers/experts
 Non-refereed – not reviewed by experts
 Pay Journals – pay to have information published
 Primary source – original articles or reports in which
researchers communicate directly the methods &
results of their study

Need to then evaluate the methods used in the study
 Secondary source – reviews, summarizes or discusses
research conducted by others
 Commentary/opinion
Quantitative & Qualitative Research
 Based on different assumptions about how to best
understand and come to know what is true


Quantitative – emphasizes numbers, measurement, deductive
logic, control & experiments
Qualitative – emphasizes natural settings, understanding,
verbal narratives, and flexible designs
Quantitative Research
 Experimental Research
 Investigators have control over 1 or more variables &
manipulate 1 factor to see if it has an impact on student
behavior
 Can be used to identify Causal relationships
True Experimental design = random assignment
 Quasi-experimental design = no random assignment
 Single Subject design = experiment with a single person or a few
individuals

Randomized Control Trials
 “Gold Standard” for evaluating an intervention’s
effectiveness
 Studies that randomly assign individuals to an
intervention group or to a control group, in order to
measure the effects of the intervention

Advantage: allows evaluation of whether the intervention
caused the outcomes, as opposed to other factors
Quantitative Research
 Non-experimental Research – no experimental
manipulation or experimental control of factors that
may influence subjects


Usually because events already occurred, or because they can’t
be manipulated
Means research can only ‘describe’ something or identify
relationships between variables; cannot determine causation
Descriptive – info. about frequency or amount of something
 Comparative – examine differences between groups on target
variable
 Correlational – investigate relationships between 2 variables
 Is there a relationship between

Single Subject Design
Example
 3 middle school
students
 Measure on-task
behavior in 15 sec.
intervals (momentary
time sampling) during
first 10 min. of class
 Intervention: Greet at
door saying students
name & positive
comment
Evaluating a Research Study
 Quantity
 One study is only one study (unless it’s a meta-analysis)
 Convergence of evidence required
 Quality
 Type of Research Design
 Sample (size & match)
 Measures (really measure important change?)
Ask a Faculty member
Collaborative Problem Solving
 Visit the website
 http://www.livesinthebalance.org/
 What do I notice?
 A canoe?
 Advertising products for purchase
 Lots of testimonials
 Little bit of research (6 studies listed)

let’s take a closer, evaluative look at the research
Collaborative Problem Solving
data from CPS website on 6/20/11







Research:
Epstein, T., & Saltzman-Benaiah, J. (2010). Parenting children with disruptive behaviors:
Evaluation of a Collaborative Problem Solving pilot program. Journal of Clinical Psychology
Practice, 27-40.
Martin, A., Krieg, H., Esposito, F., Stubbe, D., & Cardona, L. (2008). Reduction of restraint and
seclusion through Collaborative Problem Solving: A five-year, prospective inpatient study.
Psychiatric Services, 59(12), 1406-1412.
Greene, R.W., Ablon, S.A., & Martin, A. (2006). Innovations: Child Psychiatry: Use of
Collaborative Problem Solving to reduce seclusion and restraint in child and adolescent
inpatient units. Psychiatric Services, 57(5), 610-616.
Greene, R.W., Ablon, J.S., Monuteaux, M., Goring, J., Henin, A., Raezer, L., Edwards, G., &
Markey, J., & Rabbitt, S. (2004). Effectiveness of Collaborative Problem Solving in affectively
dysregulated youth with oppositional defiant disorder: Initial findings. Journal of Consulting
and Clinical Psychology, 72, 1157-1164.
Greene, R.W., Biederman, J., Zerwas, S., Monuteaux, M., Goring, J., Faraone, S.V. (2002).
Psychiatric comorbidity, family dysfunction, and social impairment in referred youth with
oppositional defiant disorder. American Journal of Psychiatry, 159, 1214-1224.
Greene, R. W., Beszterczey, S. K., Katzenstein T., Park, K., & Goring, J. (2002). Are students
with ADHD more stressful to teach? Patterns of teacher stress in an elementary school sample.
Journal of Emotional and Behavioral Disorders, 10, 27-37. These studies to
not evaluate
effectiveness of
CPS
Evaluating a Research Study
 Abstract
 Introduction & Literature Review
 Research Questions
 Method & Design
 Subjects & Settings / Measures/ Procedures
 Results
 Discussion & Conclusions
 References
Evaluating the Research studies
2004; Greene et
al
2006; Greene
, Ablon,
Martin
2008; Martin
et al
2010 –
Epstein &
Saltzmann
Subjects
Age & #
47 kids w ODD
4-12 yrs. old
3-14 yrs. old
School-age
12 Kids w ODD
Under 12 yrs.
Settings
Outpatient MH
clinic @ hospital
Inpatient Psyc
unit (13 beds)
Inpatient Psyc
unit (15 beds)
Outpatient
clinic
Procedure
Compare CPS w
parent training (PT)
group
Trained unit
staff (pre/post)
Trained unit
staff (pre/post)
Group CPS
parent training
(pre/post)
Outcome
measure
ODDRS
(unpublished rating
scale created by
Greene;
Improvement
ratings (maternal &
therapist)
Restraints &
seclusion
Restraints &
Seclusions
Eyberg Child
Beh. Inv.
Parent Stress
Index
Outcome
Improved slightly
more than PT
Reduced
Reduced
Improvement
pre to post
What does the research tell us?
 So what do we know?
 Based on 4 evaluation studies

All include children ages 12 or less (2008 study does not specify an
age range; simply ‘school age’)
2 are in inpatient psychiatric hospitals
 1 is an outpatient mental health clinic
 1 is a parent training program
 # in school settings = 0

 The research tells us nothing about the efficacy of CPS in
school settings
What does the research tell us?
 Outcome measures
 ODD Rating Scale (unpublished assessment created by the
author) & improvement ratings from parent & therapist


Reductions in restraint & seclusion (Pre/Post)


Similar scores to parent training
Is this due to student behavior change or adult behavior change?
Eyberg CBI & Parent Stress Index (Pre/Post)
 No studies directly measure changes in student
behavior
Concerns
 Only 4 research studies evaluating CPS in 4 years


2 on parent training (1 individual training & 1 group training)
2 in inpatient psyc facilities
 Make sure research you are looking at takes place in
settings that match your application

E.g. school settings v. treatment centers
 2 of 4 studies have been conducted by the author of the
program

Concern if authors are benefiting financially from sale of the
program
School-wide PBIS – Let’s compare!
 www.pbis.org
 Click on ‘Resource Catalog’
 Then ‘Literature List’
 Evidence Base for SW-PBIS
 Randomized Control Trials
Randomized Control Trials of SW-PBIS
Tier 1/ Universal SW-PBIS

Bradshaw, C. P., Mitchell, M. M., & Leaf, P. J. (2010).Examining the effects of schoolwide positive behavioral interventions and supports on student outcomes: Results from a
randomized controlled effectiveness trial in elementary schools. Journal of Positive
Behavior Interventions, 12(3), 133-148.

Bradshaw, C.,Koth, C., Bevans, K., Ialongo, N., & Leaf, P. (2008). The impact of schoolwide positive behavioral interventions and supports (PBIS) on the organizational health
of elementary schools.School Psychology Quarterly.

Bradshaw, C., Reinke, W., Brown, L., Bevans, K., & Leaf, P. (2008).Implementation of
school-wide positive behavioral interventions and supports (PBIS) in elementary schools:
Observations from a randomized trial. Education and Treatment of Children, 31, 1-26.

Horner, R. H., Sugai, G., Smolkowski, K., Eber, L., Nakasato, J., Todd, A. W., &
Esperanza, J., (2009). A randomized, wait-list controlled effectiveness trial assessing
school-wide positive behavior support in elementary schools. Journal of Positive
Behavior Interventions, 11(3), 133-144.

Sprague, J., & Biglan, A., et al (in progress).A Randomized Control Trial of SWPBS with
Middle Schools.
Meta-Analysis
 A statistical reviewing technique that provides a
quantitative summary of findings across an entire
body of research
 The results of individual studies are converted to a
standardized metric or effect size
 These scores are then aggregated across the sample
of studies to yield an overall estimate of effect size
Effect Size
 Particular attention is given to the magnitude of the
effect size



.80 = large effect size
.50 = moderate effect size
.20 = small effect size
(Cohen, 1988)
Web Resources
 What Works Clearinghouse
 IES Practice Guides
 IRIS Modules
 Meta-Analyses & Research Reviews

Many more…
Group Activity
 Review the articles and information you collected:

What are the strengths and/or limitations of the article and
resources you identified

How would your approach change now to identifying
resources?

Discuss with your group
Group Activity
 Choose 1 of each to review & present to the class:
 1 Meta-analysis from the course web page
 1 web-site from the course web page

Be prepared to do a class presentation on Monday that…
Meta-Analysis
 Briefly describes your experience reading the MetaAnalysis
 Summarize the results & finding of the meta-analysis
 Web Site Resource
 Provide an overview of the website and useful tools
 Walk people through the website and how to use feature you
found useful
