Evidence, Ethics, and the Law

Ronnie Detrich Wing Institute APBS Conference March, 2007

Overview

• Discuss the recent legal and regulatory requirements to base interventions on scientific research.

• Describe ethical requirements to rely on scientific knowledge.

• Describe what it means to be evidence-based  Definitions  Issues  Controversies

The Law

• No Child Left Behind (NCLB) requires that interventions used to improve educational performance are based on

scientific research

.

 In NCLB there are over 100 references to scientific research.

• Individuals with Disabilities Education Improvement Act [IDEIA] (2004) requires interventions that are

scientifically based instructional practices

.

The Law

• Specific requirements of IDEIA include:  Pre-service and professional development for all who work with students with disabilities to ensure such personnel have the skills and knowledge necessary to improve the academic achievement and functional performance of children with disabilities, including the use of

scientifically based instructional practices

, to the maximum extent possible .

The Law

•

Scientifically based

early reading programs, positive behavioral interventions and supports, and early intervention services to reduce the need to label children as disabled in order to address the learning and behavioral needs of such children.

The Law

• The Individualized Education Program (IEP) shall include a statement of the special education and related services and supplementary aids and services, based on

peer-reviewed research to the extent practicable

, to be provided to the child, or on behalf of the child, and a statement of the program modifications or supports for school personnel that will be provided for the child.

The Law

• In determining if a child has a specific learning disability, a local education agency may use a process that determines if a child responds to a

scientific, research-based intervention

as part of the evaluation procedures.

Ethics and Evidence-based Services

• Most national psychological and educational organizations have ethical standards requiring science-based practices to address problems.

 American Psychological Association Ethical Standard 2.04: 

Psychologists’ work is based on the established scientific and professional knowledge of the discipline.

Ethics and Evidence-based Services

• National Association of School Psychologists  Standard III F 4.



School psychology faculty members and clinical or field supervisors uphold recognized standards of the profession by providing training related to high quality, responsible, and research-based school psychology services.

Ethics and Evidence-based Services

• National Association of School Psychologists  Standard IV 4.



School psychologists use assessment techniques, counseling and therapy procedures, consultation techniques, and other direct and indirect service methods that the profession considers to be responsible, research-based practice .

Ethics and Evidence-based Services

• Behavior Analysis Certification Board  Standard 2.09a



The behavior analyst always has the responsibility to recommend scientifically supported, most effective treatment procedures . Effective treatment procedures have been validated as having both long-term and short-term benefits to clients and society.

 Standard 2.09b



Clients have a right to effective treatment (i.e., based on the research literature and adapted to the individual client).

Another Ethical Responsibility

• Special education services are largely funded through public dollars (taxpayers dollars).

• There is an implicit assumption that the money will be spent for the public good.

• A fiduciary responsibility exists when one person or organization is charged with managing another person’s money.

Another Ethical Responsibility

• The primary responsibility of a fiduciary is to act solely for the benefit of the other party.

• Being a fiduciary carries the weight of ethical conduct.

• Those of us charged with providing special education services have a fiduciary responsibility to assure that the taxpayers are receiving the greatest possible return on their investment.

How Do We Meet Our Fiduciary Responsibility?

• Interventions that have an evidence base are more likely to produce positive effects for students.

• Requires that we use the intervention that has a higher probability of success, i.e. evidence-based intervention.

How Do We Meet our Fiduciary Responsibility?

• It is unknown what impact non-evidence-based interventions are likely to have.

 Using a non-evidence-based intervention when there are evidence-based interventions available constitutes unethical practice.

 If we want to use a non-evidence based intervention then it should be considered research and all of the safe-guards afforded research participants and their families should be in place.

 Conducting research with tax-dollars provided for education may constitute a violation of our fiduciary responsibility.

Becoming Evidence-based

• Clearly, the intent of Congress, the U.S. Department of Education, and Office of Special Education Programs is to rely on interventions that have a scientific basis.

• Professional organizations place great value on scientific knowledge.

• What does it mean to be evidence-based?

What is Evidence-based Practice?

• Sackett et al (2000) defined evidence-based practice in medicine as: “

the integration of best research evidence with clinical expertise, and patient values.”

• EBP is a decision-making approach that places emphasis on evidence to:  guide decisions about which interventions to use.

 evaluate the effects of an intervention.

What is Evidence-based Practice?

• Ultimately, EBP is a consumer protection issue.

 Assumes that evidence-based interventions are more likely to be effective than interventions that are not evidence-based.

 By validating interventions as evidence-based there is the implication that there are standards for reviewing interventions.

 Standards should be transparent.

• EBP is more than identifying evidence-based interventions.

Evidence-based Practice

Identifying Evidence-based Interventions

• Controversies to identifying evidence-based interventions.

 There is no consensus about what constitutes evidence.

 NCLB permits both quantitative and qualitative evidence without specifying the types of questions that each approach best answers.

 In this context, we are most often concerned with evidence that establishes a causal relation between an intervention and a class of social or academic behaviors.

Identifying Evidence-based Interventions

• Even if we accept causal demonstrations to be evidence, we have no consensus.

 Randomized Clinical Trials have become the “gold standard.”  May pose ethical and logistical challenges in special education.

 There is controversy about the status of single participant designs.

 Most frequently criticized on the basis of external validity.

Identifying Evidence-based Interventions

• Identification is more than finding a study to support an intervention.

 Involves an evaluation of the body of knowledge about an intervention.

 Standards of evidence specify:  the quantity of evidence required to validate an intervention.

 the quality of evidence necessary to validate an intervention as evidence-based.

 There are no agreed upon standards.

 No single resource for decision makers.

 May result in criteria other than evidence to influence decisions.

Identifying Evidence-based Interventions

• Two approaches to establishing standards  Threshold approach:  Evidence must be of a specific quantity and quality before intervention is considered evidence-based.

 Risk is that effective practices will be ruled out because they fail to meet evidentiary standards.

 Hierarchy of evidence approach:  Best available evidence approach.  Strength of evidence falls along a continuum with each level having differential standards.

 Risk including practices that are not effective because the criterion for inclusion is less rigorous.

Identifying Evidence-based Interventions

• Several organizations have established standards but there is limited agreement among them.

 What Works Clearinghouse  National Autism Center (proposed)  Society for Prevention Research  Task Force on Evidence-based Interventions in School Psychology  Council for Exceptional Children (proposed)  It is possible for an intervention to meet one standard but not a second.

 The multiple organizations with standards leaves the consumer to know which organization validated an intervention.

Identification of Evidence-based Interventions

• Types of Research:  Efficacy trials: Intervention demonstrated to be effective when implemented under optimum conditions.

 Primarily designed to demonstrate impact of an independent variable.

 Effectiveness trials: Interventions implemented under more typical practice conditions.

 Often interventions effects compromised when taken to scale.

 Much of the published research is considered efficacy demonstrations.

 Even if meets evidentiary standards to be identified as evidence-based may not translate into effective practice when implemented in practice settings.

Implementing Evidence-based Interventions

Where Good Interventions Go to Die

• Is identifying evidence-based interventions sufficient to meet regulatory and ethical requirements?

 Intent of both legal and ethical guidelines is to have positive impact.

 Evidence-based interventions are assumed to give to give us that chance.

• Identification is necessary but not sufficient to assure that intervention will be effective.

• Must address complex issues associated with implementation.

Issues of Implementation

• Dissemination:

getting the evidence into the hand of decision-makers

.

• Selection:

assuring that evidence influences decisions about interventions.

• Initial Implementation:

assuring that those responsible for implementation have necessary resources to be successful.

• Sustainability:

assuring effective programs survive.

Implementing Evidence-based Interventions

• Non-science based issues influence selection of intervention:  Personal experience  Expert opinion  Cost impacts adoption  Less effective but cheaper interventions may be adopted.

 Effort  Contextual fit.

 Complexity of intervention.

 Training.

Implementing Evidence-based Interventions

Contextual Fit

• Contextual Fit:

the degree to which an intervention matches the culture, training, and resources of a particular setting.

 These characteristics of a setting can be measured.

 Degree of contextual fit may moderate the impact of an intervention.

Implementing Evidence-based Interventions

Contextual Fit

• Logically it would seem to make sense to always implement intervention that produces greatest impact.

 There may be exceptions:  If high impact intervention requires great resources, specialized training, and is very different from current practices it may not be implemented with integrity.

 May be better to implement effective but lower impact intervention that is better contextual fit and will be implemented with greater integrity.

Implementing Evidence-based Interventions

Contextual Fit

• Adoption or Accommodation  Adoption: Implementing intervention as evaluated to be effective.

 Assures evidence-based intervention is being implemented.

 Accommodation: adjusting intervention to meet local circumstances.

 May result in intervention no longer being evidence-based.

 May increase implementation with integrity.

Evidence-based and Empirically-supported

Contextual Fit

• As we make accommodations to an intervention the certainty of it being evidence-based is compromised.

• An alternative approach is to assure that the accommodations are empirically-supported.

 The accommodation is based on well established principles of behavior  If the accommodation is based on principles of reinforcement then is empirically-supported.

 Not evidence-based because the intervention has not been evaluated as a practice.

Implementing Evidence-based Interventions

Complexity

• Level of precision may increase complexity  Be as precise as necessary but no more.

 Catch’em being good  Good behavior game  Individualized intervention plan • Increase in complexity may represent increase in effort.

Implementing Evidence-based Interventions

Training

• Implementing a new intervention requires training  Presents significant financial and logistical challenges.

 How is everyone to be trained?

 Who will provide the training?

 How will acquisition be assured?

 How will maintenance be assured?

 How will new implementers be trained after initial training?

Implementing Evidence-based Interventions

Dissemination

• Dissemination: getting information into hands of practice-based decision makers.

 Journals are poor means for dissemination.

 Not typically read by decision makers  Decision makers not necessarily qualified to interpret results  Often lack sufficient detail for replication in a practice setting.

 Clearinghouses are in their infancy.

 Often do not have the information a decision maker is seeking.

Implementing Evidence-based Interventions

• Often the evidence for a particular set of problems is inadequate for identifying an evidence-based intervention.

 What becomes the basis for decision making?

 An alternative to evidence-based:  Evidence-based: meets well defined standards.

 Empirically supported: based on scientific principles of behavior.

 When no evidence-based intervention then build intervention with exclusively with principles from science of human behavior.

» Not as strong as evidence-based.

Implementing Evidence-based Interventions

Sustainability

• If intervention is effective then it is desirable to assure sustainability of the program.

 Defining features of sustainable programs:  Maintains over time.

 Maintains across generations of practitioners.

 Supported with existing resources of system.

 There is an emerging science of sustainability.

• The larger the scale of implementation the greater the degree of complexity.

 The science of large scale implementation is not well established .

Evaluating Evidence-based Interventions

Progress Monitoring

• Implementation of evidence-based intervention does not assure success.

 Necessary to evaluate impact in local context.

 No intervention will be effective for all students.

 Cannot predict who will benefit.

 Progress monitoring is practice-based evidence about evidence-based practices.

 Consistent with legal requirements and ethical standards.

Ethical Standards and Progress Monitoring

• National Association of School Psychologists  Standard IV C 1b.



Decision-making related to assessment and subsequent interventions is primarily data-based .

 Standard IV 6.



School psychologists develop interventions that are appropriate to the presenting problems and are consistent with the data collected. They modify or terminate the treatment plan when the data indicate the plan is not achieving the desired goals.

Ethical Standards and Progress Monitoring

• Behavior Analysis Certification Board  Standard 4.04



The behavior analyst collects data or asks the client, client surrogate, or designated other to collect data needed to assess progress within the program.

 Standard 4.05



The behavior analyst modifies the program on the basis of data.

Legal Requirements for Progress Monitoring

• Fundamental to IEP process.

 Must report on same schedule that grades are reported in general education.

• Response to Intervention is accepted as alternative means for determining eligibility for Learning Disability classification.

 Progress monitoring is the heart of RTI.

 All students routinely and systematically monitored to assure adequate progress is occurring.

Evaluating Evidence-based Interventions

• Progress monitoring is a systems level intervention.

 Contingencies must be in place to assure:  Data are collected  Data are reviewed  Decisions are based on the data.

 If contingencies are not in place, response effort associated with data collection will compromise data based decision making.

Evaluating Evidence-based Interventions

• Curriculum based measurement is a powerful means for evaluating impact of academic interventions.

 Scores on CBM correlated with scores on high stakes test.

 Can be used to predict how students will perform on state-wide tests.

Evaluating Evidence-based Interventions

• Monitoring effects of intervention of students is necessary but it is also necessary to systematically assess accuracy of implementation (treatment integrity).

 Implementation without integrity results in an unspecified intervention being implemented for which there is no evidence-base to justify the intervention.

 If student is not benefiting from intervention and there is poor integrity then no conclusions can be drawn about the effect of the intervention.

 If integrity is low, increase integrity before evaluating impact.

Where Are We?

• Legal and ethical requirements encourage evidence-based practice.

• What is the state of the field?

Where are We?

• Kazdin (2000) identified 550 named interventions for children and adolescents.

 A very small number of these interventions have been empirically evaluated.

 Of those that have been evaluated, the large majority of them are behavioral or cognitive-behavioral.

 Evidence-based interventions are less likely to be used than interventions for which there is no evidence or there is evidence about lack of impact (Kazdin, 2000).

 In many instances practitioners are not aware of evidence-based interventions (Kratochwill, Albers & Shernoff, 2004).

How Do We Achieve the Reality of Evidence-based Practice?

• The legal and ethical requirements to implement evidence-based interventions requires system level changes.

 Systems requirements:  Process for selecting interventions.

 Process for training and maintenance.

 Process for progress monitoring for both treatment integrity and student performance.

 These changes to the system will not occur rapidly.

 Behavioral systems perspective can facilitate change process.

 Gilbert (1996) Human Competence: Engineering Worthy Performance

Thank you

• Copies of presentation can be obtained at www.winginstitute.org