Transcript The Ethical and Legal Basis for Evidence

The Ethical and Legal Basis for Evidencebased Education: Implications for the
Profession
Ronnie Detrich
Wing Institute
TED Conference
November, 2008
Dallas, Texas
Overview
• Discuss the recent legal and regulatory
requirements to base educational
interventions on scientific research.
• Describe ethical requirements to rely on
scientific knowledge.
• Describe what it means to be evidence-based
 Definitions
 Issues
 Controversies
• Appraisal of current status and future
directions.
The Legal Basis
• No Child Left Behind (NCLB): Interventions
used to improve educational performance are
based on scientific research.
 Over 100 references to scientific research in
NCLB.
• Individuals with Disabilities Education
Improvement Act [IDEIA] (2004):
Interventions are scientifically based
instructional practices.
The Legal Basis
• Specific requirements of IDEIA include:
 Pre-service and professional development to
improve the academic achievement and functional
performance of children with disabilities, including
the use of scientifically based instructional
practices, to the maximum extent possible.
The Legal Basis
• Scientifically based early reading programs,
positive behavioral interventions and
supports, and early intervention services to
reduce the need to label children as disabled
in order to address the learning and
behavioral needs of such children.
The Legal Basis
• The Individualized Education Program (IEP)
services based on peer-reviewed research to
the extent practicable.
The Legal Basis
• In determining if a child has a specific
learning disability, a local education agency
may use a process that determines if a child
responds to a scientific, research-based
intervention as part of the evaluation
procedures.
The Ethical Basis
• Most national psychological and educational
organizations have ethical standards
requiring science-based practices to address
problems.
 American Psychological Association Ethical
Standard 2.04:
 Psychologists’ work is based on the established scientific
and professional knowledge of the discipline.
The Ethical Basis
• National Association of School Psychologists
 Standard III F 4.
 School psychology faculty members and clinical or field
supervisors uphold recognized standards of the
profession by providing training related to high quality,
responsible, and research-based school psychology
services.
The Ethical Basis
• National Association of School Psychologists

Standard IV 4.
 School psychologists use assessment techniques,
counseling and therapy procedures, consultation
techniques, and other direct and indirect service methods
that the profession considers to be responsible,
research-based practice.
The Ethical Basis
• Behavior Analyst Certification Board
 Standard 2.09a
 The behavior analyst always has the responsibility to
recommend scientifically supported, most effective
treatment procedures. Effective treatment procedures
have been validated as having both long-term and shortterm benefits to clients and society.
 Standard 2.09b
 Clients have a right to effective treatment (i.e., based on
the research literature and adapted to the individual
client).
Another Ethical Responsibility
• Education services are largely funded
through public dollars (taxpayers dollars).
• There is an implicit assumption that the
money will be spent for the public good.
• A fiduciary responsibility exists when one
person or organization is charged with
managing another person’s money.
Another Ethical Responsibility
Fiduciary
Act solely for the
benefit of the
other party
Carries the
weight of ethical
conduct
Assure that the
taxpayers are
receiving the
greatest possible
return on their
investment.
How Do We Meet Our Fiduciary
Responsibility?
• Interventions that have an evidence base are
more likely to produce positive effects for
students.
 Does not assure positive outcomes but increases
the probability.
• The impact of a non-evidence based
intervention is unknown.
How Do We Meet our Fiduciary
Responsibility?
• Implications:
 Using a non-evidence-based intervention when
there are evidence-based interventions available
constitutes unethical practice.
 Use of a non-evidence based intervention should
be considered research.
 All of the safe-guards afforded research participants and
their families should be in place.
 Conducting research with tax-dollars provided for
education services may constitute a violation of
our fiduciary responsibility.
Becoming Evidence-based
• Clearly, the intent of Congress, the U.S.
Department of Education, and Office of
Special Education Programs is to rely on
interventions that have a scientific basis.
• Professional organizations place great value
on scientific knowledge.
• What does it mean to be evidence-based?
What is Evidence-based Practice?
• At its core the EBP movement is a consumer
protection movement.
 It is not about science per se.
 It is a policy to use science for the benefit of
consumers.
 “The ultimate goal of the ‘evidence-based
movement’ is to make better use of research
findings in typical service settings, to benefit
consumers and society….” (Fixsen, 2008)
What is Evidence-based Practice?
• Evidence-based practice has its roots in
medicine.
 Movement has spread across major disciplines in
human services:
 Psychology
 School Psychology
 Social Work
 Speech Pathology
 Occupational Therapy
What Is Evidence-based Practice?
Professional
Judgment
Best available
evidence
Client Values
Sackett et al (2000)
Values
Professional
Available
Judgment
Evidence
• EBP isBest
aClient
decision-making
approach that
places emphasis on evidence to:
 guide decisions about which interventions to use;
 evaluate the effects of an intervention.
Phases of Evidence-based
Intervention
Identify
Identify
Evidence-based
Intervention
Evaluate
Evaluate
Implement
Implement
Identify
What is Evidence-based Education?
• The term “evidence-based” has become
ubiquitous in last decade.
 There is no consensus about what it means.
 At issue is what counts as evidence.
 Federal definition emphasizes experimental
methods.
 Preference for randomized trials.
 Definition has been criticized as being positivistic.
Identify
What Counts as Evidence?
• Ultimately, this depends on the question
being asked.
 Qualitative methods are best for some questions.
• In EBP the goal is to identify causal relations
between interventions and outcomes.
 Experimental methods do this best.
Identify
What Counts as Evidence?
• Even if we accept causal demonstrations to
be evidence, we have no consensus.
 Randomized Clinical Trials (RCT) have become
the “gold standard.”
 There is controversy about the status of single
subject designs.
 Most frequently criticized on the basis of external validity.
Identify
How Are Evidence-based Interventions
Identified?
• Identification is more than finding a study to
support an intervention.
• Identification involves distilling a body of
knowledge to determine the strength of
evidence.
Identify
How Are Evidence-based Interventions
Identified?
• Distillation requires standards of evidence for
reviewing the literature.
 Standards specify:
 the quantity of evidence
 the quality of evidence
Identify
Continua of Evidence
Quantity of the Evidence
Meta-analysis
(systematic review)
Repeated Systematic
Measures
Single Case Replication
(Direct and Parametric)
Threshold
Convergent Evidence
of
Evidence
Quality of the Evidence
Current “Gold Standard”
High Quality
Randomized Controlled Trial
Single Case Designs
Semi-Randomized Trials
Well-conducted
Clinical Studies
Uncontrolled Studies
Expert Opinion
Various Investigations
Single Study
General Consensus
Personal Observation
Janet Twyman, 2007
Identify
How Are Evidence-based Interventions
Identified?
• Two approaches to validating interventions
 Threshold approach:
 Evidence must be of a specific quantity and quality
before an intervention is considered evidence-based.
 Hierarchy of evidence approach:
 Strength of evidence falls along a continuum with each
level having differential standards.
Identify
How Are Evidence-based Interventions
Identified?
• There are no agreed upon standards.
 It is possible for an intervention to be evidencebased using one set of standards and to fail to
meet evidence standards using an alternative set.
 Difficult for consumers and decision makers to sort out
the competing claims about what is evidence-based.
Identify
Identify
Evidence-based
Intervention
Evidence-based
Intervention
Identify
Ineffective Effective
Effective
Ineffective
Assessed Effectiveness
Actual Effectiveness
Effective
Effective
Ineffective
Ineffective
True
Positive
Most
likely
False
withPositive
hierarchy
approach
Most
likely
False
with
threshold
Negative
approach
True
Negative
Identify
Choosing Between False Positives and
False Negatives
• At this stage, it is better to have more false
positives than false negatives.
False Negatives:
Effective interventions will
not be selected for
implementation.
As a consequence, less
likely to determine that they
are actually effective.
False Positives:
Progress monitoring
will identify
interventions that are
not effective.
Why Do We Need Evidence-based
Education?
550 named interventions for children and adolescents
Kazdin (2000)
Empirically evaluated
Behavioral
Cognitivebehavioral
Evidence-based interventions are less likely to be used than interventions
for which there is no evidence or there is evidence about lack of impact.
Are We Training Educators to be
Evidence-based?
Survey of School Psychology Directors of Training
(Shernoff, Kratochwill, & Stoiber, 2003)
Evidence-based interventions
29%
Knowledge
41% programs
training
What Are Teachers Being Taught
About Science of Reading?
40
35
30
Percent
25
20
15
10
5
0
0
National Council on Teacher Quality , 2006
1
2
3
# of Re ading Ele m e nts Taught
4
5
Implement
Implementing Evidence-based
Interventions
Where Good Interventions Go to Die
• Intent of both legal and ethical guidelines is to
have positive impact.
 Evidence-based interventions are assumed to give
to give us that chance.
• Identification is necessary but not sufficient to
assure that intervention will be effective.
• Must address complex issues associated with
implementation.
Implement
Implementing Evidence-based
Interventions
• Implementation is where the research to
practice gap is most evident.
• Many innovations in education have very
short life spans.
 18 mos. (Latham)
 Most often a result of poor implementation.
Implementation is Fundamental
100
80
60
80% of initiatives
ended within 2 years
40
90% of initiatives
ended within 4 years
20
0
2000
Data from Center for
Comprehensive
School Reform
2002
2004
Implement
Well Tested Programs Often Fail Large
Scale Implementation
• Elliott & Mihalic (2004) review Blueprint
Model Programs (violence prevention and
drug prevention programs) replication in
community settings.
 Programs reviewed across 5 dimensions
 Site selection
 Training
 Technical assistance
 Fidelity
 Sustainability
Implement
Keys to Implementation
• Critical elements in site readiness
 Well connected local champion
 Strong administrative support
 Formal organizational commitments
 Formal organizational staffing stability
 Up front commitment of necessary resources
 Program credibility within the community
 Program sustained by the existing operational
budget
Implement
Keys to Implementation
• Critical elements of training
 Adhere to requirements for training, skills, and
education.
 Hire all staff before scheduling training.
 Encourage administrators to attend training.
 Plan and budget for staff turnover.
 Implement program immediately after training.
Implement
Keys to Implementation
• Critical elements of Technical Assistance
 Proactive plan for technical assistance.
• Critical elements of Fidelity
 Monitor fidelity.
• Critical elements of Sustainability
 Function of how well other dimensions are
implemented.
Implement
Implementing Evidence-based
Interventions
Dimensions of Implementation
• Contextual fit
• Complexity of intervention
Implement
Implementing Evidence-based
Interventions
Contextual Fit
• Contextual Fit: the degree to which an
intervention matches the culture, training, and
resources of a particular setting.
 These characteristics of a setting can be
measured.
 Degree of contextual fit may moderate the impact
of an intervention.
Implement
Implementing Evidence-based
Interventions
Contextual Fit
• Adoption or Accommodation
 Adoption: Implementing intervention as evaluated
to be effective.
 Assures intervention is evidence-based.
 Does not assure implementation.
 Accommodation: adjusting intervention to meet
local circumstances.
 May result in intervention no longer being evidencebased.
 May increase implementation with integrity.
Implement
Implementing Evidence-based
Interventions
Contextual Fit
• Logically it would seem to make sense to
always implement intervention that produces
greatest impact.
 There may be exceptions:
 If high impact intervention requires great resources,
specialized training, and is very different from current
practices it may not be implemented with integrity.
 May be better to implement effective but lower impact
intervention that is better contextual fit and will be
implemented with greater integrity.
Implement
Implementing Evidence-based
Interventions
Complexity
• Level of precision may increase complexity.
Complexity
 Be as precise as necessary but no more.
Good
Behavior
Game
Catch’em
being
good
Precision
Individualized
intervention
plan
Evaluate
Evaluating Evidence-based Interventions
Progress Monitoring
• Implementation of evidence-based
intervention does not assure success.
 Necessary to evaluate impact in local context.
 No intervention will be effective for all students.
 Cannot predict who will benefit.
 Progress monitoring is practice-based evidence
about evidence-based practices.
 Consistent with legal requirements and ethical
standards.
Evaluate
Ethical Standards and Progress
Monitoring
• National Association of School Psychologists
 Standard IV C 1b.
 Decision-making related to assessment and subsequent
interventions is primarily data-based.
 Standard IV 6.
 School psychologists develop interventions that are
appropriate to the presenting problems and are
consistent with the data collected. They modify or
terminate the treatment plan when the data indicate the
plan is not achieving the desired goals.
Evaluate
Ethical Standards and Progress
Monitoring
• Behavior Analyst Certification Board
 Standard 4.04
 The behavior analyst collects data or asks the client,
client-surrogate, or designated other to collect data
needed to assess progress within the program.
 Standard 4.05
 The behavior analyst modifies the program on the basis
of data.
Evaluate
Legal Requirements for Progress
Monitoring
• Fundamental to IEP process.
• Response to Intervention is accepted as
alternative means for determining eligibility
for Learning Disability classification.
 Progress monitoring is the heart of RTI.
 All students routinely and systematically monitored to
assure adequate progress is occurring.
Evaluate
Evaluating Evidence-based Interventions
• Progress monitoring is a systems level
intervention.
 Systems must be in place to assure:
 Data are collected
 Data are reviewed
 Decisions are based on the data
 If systems are not in place, response effort
associated with data collection will compromise
data-based decision making.
Evaluate
Progress Monitoring as an Intervention
• Progress monitoring 2-5/week in math and
reading:
 4 times as effective as 10% increase in per pupil
spending;
 6 times as effective as voucher programs;
 64 times as effective as charter schools;
 6 times as effective as increased accountability.
(Yeh, 2007)
Evidence-based Education and Treatment
Integrity
• Progress monitoring allows data based
decision making about effects of an
intervention.
 It is impossible to make informed decisions
without knowing how well the intervention was
implemented.
Integrity
High
Low
Low
High
Outcome
Negative
Positive
Negative
Positive
Continue
Intervention
Unknown reason
Change
Intervention
Unknown reason
• Other life changes?
• Intervention problem?
• Unknown
intervention?
• Implementation problem?
• Intervention is
effective?
Where Are We?
From a university in the U.S.
Where are We?
• Being evidence-based is more than a good
idea, it is the law and it is ethical conduct;
however, it is not as easy as it might seem.
• Lack of consensus about evidence may do
harm to consumers.
• The research to practice gap limits the impact
of evidence-based education.
 The science of implementation and sustainability
is in its infancy.
Where are We?
• Pre-service training should change to reflect
current policy.
 Changes in both method of training and content.
Effects of Training
OUTCOMES
(% of Participants who demonstrate knowledge, demonstrate
new skills in a training setting,
and use new skills in the classroom)
Knowledge
Skill
Demonstration
Use in the
Classroom
Theory and
Discussion
10%
5%
0%
..+Demonstration
in Training
30%
20%
0%
…+ Practice &
Feedback in
Training
60%
60%
5%
…+ Coaching in
Classroom
95%
95%
95%
TRAINING
COMPONENTS
Joyce and Showers, 2002
Basis for Choosing Treatment
Szatmari (2004)
Treatment
Values
Evidence
Do Nothing
Unethical
Clinical
Paralysis
None
Do Nothing
Unethical
Clinical
Paralysis
None
Toss a Coin
Unethical
in light of
evidence
None
Do Nothing
Unethical
Clinical
Paralysis
None
Toss a Coin
Unethical
in light of
evidence
None
Training
Outdated
None
Current
Perhaps
some
Do Nothing
Unethical
Clinical
Paralysis
None
Toss a Coin
Unethical
in light of
evidence
None
Training
Outdated
None
Etiology
Difficult
Limited
Do Nothing
Unethical
Clinical
Paralysis
None
Toss a Coin
Unethical
in light of
evidence
None
Training
Outdated
None
Etiology
Difficult
Limited
ABA
Not very
humane
Effective
Robust
Do Nothing
Toss a Coin
Unethical
Clinical
Paralysis
Unethical
in light of
evidence
None
ABA
Not very
humane
Robust
None
Training
Outdated
None
Etiology
Difficult
Limited
Developmental
sociocognitive
Highly
preferred
None
Yet
To be Ethical:
Inform Parents of Options
Do Nothing
Toss a Coin
Unethical
Clinical
Paralysis
Unethical
in light of
evidence
None
ABA
Not very
humane
Robust
None
Training
Outdated
Etiology
None
Developmental
sociocognitive
Highly
preferred
None
yet
Difficult
limited
Thank you
Copy of this presentation may be downloaded
at
www.winginstitute.org