Transcript Feeding Information Learned from Adaptations Back into

Feeding Information Learned from
Adaptations Back into Program Design
William B. Hansen
Dana Bishop
Linda Dusenbury
Melinda Pankratz
Jordan Albritton
Lauren Albritton
Joann Strack
Acknowledgments
Funding: R01DA024639, R44DA020954
Conflict of Interest: Tanglewood Research
owns and markets All Stars. Dr. Hansen is the
President and primary owner
of Tanglewood Research
Abstract (1)
With the widespread dissemination of evidencebased programs, quality of implementation has
become a central focus of research. Among
other topics, the role played by adaptations
made during program delivery is of increasing
concern In this presentation we will discuss
how using information gleaned from observing
how teachers adapt a program can be used to
improve program design.
Abstract (2)
We coded video recordings made by nine teachers,
each of whom taught All Stars over a three-year
period. We categorized changes as (1) changes to
the structure of the program, (2) changes in
instructions given to students, (3) added steps, (4)
added or altered questions, (5) added examples or
stories, (6) new messages about norms, (7) added
motivational messages, (8) the introduction of
concepts not called for by the program, and (9)
interpretations of interest. Each adaptation was
assigned a valence score ranging from –2 to +2
based its potential to further or to interfere with
program goals and objectives.
Abstract (3)
All teachers adapted, making an average of 7.2
adaptations per session. Five teachers had
overall negative average valence ratings,
ranging from –0.03 to –0.35. Four teachers had
overall positive average valence ratings,
ranging from 0.24 to 0.56.
Abstract (4)
On average, the most positively rated adaptations had
to do with added motivational statements (mean
valence 0.61). Steps that teachers added and
questions that teachers added were positively rated
on average (valence ratings of 0.27 and 0.15,
respectively). Further, when teachers made minor
changes that reflected their style of teaching, these
were rated positively (0.13) on average. Changing
instructions (–0.26), structure (–0.21), and introducing
new concepts (–0.17) were all viewed on average as
detrimental.
Abstract (5)
Early in the observational process, it became
clear that the observed adaptations had the
potential to inform changes in the design of the
program. For each session, a complete listing
of adaptations was produced. Positive
adaptations offered clues about new program
elements. Negative adaptations were used to
identify weaknesses that needed correction.
Abstract (6)
As a result of analyzing the spectrum of adaptations,
the following general improvements were
incorporated into the curriculum. (1) “Do not do”
warnings were added in places where negative
adaptations had been commonly observed. (2) “Point
to make” boxes were highlighted to help ensure key
concepts were emphasized and appropriate
conclusions were reached. (3) Activities prone to
negative adaptations were simplified (usually
shortened) and instructions clarified. (4) Positive
adaptations, such as insightful questions and added
motivational messages, were included. Specific
examples of adaptations that have resulted in
changes to the design of All Stars will be reviewed.
Traditional Model of Program
Development

Developer has a model

From theory/paradigm

From experience/research

Developer creates a program manual

Program is tested in pilot and field trials

If successful, the program makes it on the list

Developer markets the program
The Challenge


Inertia

The list

Reputation

The cost of change
Ignorance


What should be changed?
Confidence|Arrogance

The developer knows a lot

What do implementers know?
The Reality

No one gets it right the 1st, 2nd, 3rd, . . . time

Developers can be blinded by early success

Implementers know a lot

The world is evolving

–
Youth culture
–
Technology
What we know is expanding
The Solution
Detailed feedback about performance that
guides continuous improvement
The Strategy


Observed 324 video recordings

9 teachers

3 cycles of teaching

12 sessions of All Stars
Coded fidelity

Adherence

Quality of delivery

Adaptations

Interpretations of Interest
Adherence: What Was Done
Adherence: How Well It Was Done
How well were activity objectives achieved? (1-4 scale)
How Well It Was Done by Teacher
Quality of Teaching
Rate the overall quality of teaching (1-5 scale)
Quality of Teaching by Teacher
Adaptations: Number by Type
Total = 1,815
Adaptations: Valence by Type
Adaptation Valence by Teacher
Interpretations of Interest
Total = 445
What Does This Mean for Redesign?



Sessions vary in strength
–
Objectives need to be clarified
–
All sessions can be streamlined
Teachers vary in their delivery
–
Success is to be emulated
–
Failure is to be analyzed for clues
Adaptations vary in their usefulness
–
Motivational messages are to be encouraged
–
Negative norm messages are to be avoided
–
More questions are to be added
Useful Strategies
Extensive notes about adaptations and
interpretations
Each adaptation/interpretation was described
Time on the DVD was documented
Consensus about problems and solutions
Simplified and more directive language
Elimination of extraneous steps
Increased use of symbols and color
From All Stars Core 2.9
Adaptations Observed
From All Stars Core 3.1
More From All Stars Core 3.1
Final Words
Contact:
[email protected]