Transcript Designing an International Project on the Effectiveness of

An International Project on
the Effectiveness of
Psychotherapy and
Psychotherapy Training
(IPEPPT): Research
Framework and Protocols
Robert Elliott
University of Toledo
Research-Practice Gap in Era of
Evidence-Based Mental Health


Numerous contemporary attempts to link
research & practice in psychotherapy
Top-down solutions:



Based on:



Empirically-supported treatments
Evidence-Based Practice
Randomized Clinical Trials research model
Therapist-as-research-consumer model
Results have been mixed
Research-Practice Integration
as a Two-way, Dialectic Process




Success is more likely if we add a more
integrative, bottom-up strategy
Building on Mental Health Services/
Therapy Effectiveness paradigm
Existing RCT research makes space for
grass-roots-based research in real world
practice and training settings
=Practice-based Evidence
Example: Practitioner
Research Networks (PRNs)



USA: Pennsylvania (Ragusea, Borkovec,
Castonguay)
UK: National Health Service CORE
research team (Barkham, Evans et al.)
Latest trend: Practice-based research in
training clinics and centers (e.g.,
Castonguay et al.)
Practice-Based Therapy
Research in Training Sites




Training site research movement: USA, Europe
Research on psychotherapy process/outcome is
essential for understanding and improving
psychotherapy practice in all orientations
Being able to use and carry out research is an
important aspect of therapist competence
Best way to learn therapy research methods:



Do research during basic therapy training
Primary professional socialization process
Create habits that carry over into later practice
Principles for Practice-based
Research
1)
2)
3)
4)
5)
6)
Make research relevant to actual practice of therapy
Use methods that support therapy rather than interfere with it
Actively and continuously involve therapists in selection of
research questions and methods
Include inexpensive and easy-to-use instruments of key
elements (therapeutic alliance, client problem severity)
Encourage variety of research methods (qualitative &
quantitative; group & single-case)
Create research networks of training sites using similar, pantheoretical instruments
International Project on the
Effectiveness of Psychotherapy and
Psychotherapy Training (IPEPPT)

Formally initiated, June 2004, by:



Italian Coordinamento Nazionale Scuole di Psicoterapia
(CNSP; >5,000 therapists)
Italian Federation of Psychotherapy Associations
(FIAP; 21 psychotherapy associations: >10,000
therapists)
General Goal: To improve psychotherapy and
psychotherapy training in a broad range of
theoretical approaches by encouraging systematic
research in therapy training institutes and
university-based training clinics.
IPEPPT General Scientific
Steering Committee







Robert Elliott, Scientific Director (University of ToledoUSA)
Alberto Zucconi, Coordinator (University of Siena-Italy)
David Orlinsky (University of Chicago-USA)
Franz Caspar (University of Freiburg)
Louis Castonguay (Pennsylvania State University-USA)
Glenys Parry (University of Sheffield-UK)
Bernhard Strauss (Friedrich Schiller University JenaGermany)
IPEPPT: Current Status





Still in formation stage
Not a single study
The “Project” = Promoting practice-based
research in Europe, North American and
elsewhere
Finding partners
Creating/finding tools

E.g., conceptual/organizing concepts
IPEPPT Specific Objectives

1. To construct a list of agreed-upon general
pantheoretical recommendations for evaluating:




2. To facilitate the development of specific treatment
and training outcome protocols for particular:
Therapy approaches (e.g., Systemic therapy)
 Client populations (e.g., people living with schizophrenia)
 Linguistic/national groups (e.g., Italy)
3. To facilitate national/international collaborations


Key aspects of therapy, especially in training centers
Key aspects of therapy training outcome
Not a “Core Battery”
IPEPPT Draft Research
Framework



Such a project requires a guiding conceptual
framework for determining what to measure
and how to measure it
Work-in-progress
8 measurement domains:


4 Research themes
2 Levels (Star design)
Framework: Eight Therapy Measurement
Domains, with examples of key concepts
Research Theme
A. General/
Pantheoretical
B. TreatmentSpecific
I. Therapy
Outcome
e.g., general problem e.g., theory-based
severity
dysfunctional
processes
II. Therapy
Process
III. Client/Therapist Background
e.g., therapeutic
alliance
e.g., demographics
IV. Training
Outcome
e.g., productive vs.
e.g., therapist skill
unproductive practice development
pattern
e.g., therapist
techniques
e.g., preference for
type of therapy
Structure: (1) “Star” Design

Main body of the star = General outcome/
process protocol



Shared by all orientations (General/
Pantheoretical)
Provides common metric
Star rays = Specialized protocols for
different therapy approaches and different
countries (Treatment/Population/Language
Specific)
“Star” Design for Sample Concepts within
Therapy Outcome Domain for Studies of Four
Different Therapies
CBT
Dysfunctional Attitudes
Self-Ideal Discrepancy
Target Problems
Implicit
Cognitive
Biases
Experiential
Experiential
Access
Self-Esteem
General problem
severity
Interpersonal/
relational issues
CCRT
Change
Maturity of
Defenses
Psychodynamic
Level of Object
Relations
Qualitative
perceptions of
change
Relational
Satisfaction
Family
Environment
Interpersonal Empathy
Family/
Couples
Structure: (2) Nested Priority
Lists



Not a single “core battery”
Allow flexibility while encouraging
consistency within & across approaches
Three Levels of Priorities:
1)
2)
3)
Measurement domains are prioritized
Within each measurement domain, key concepts
are ranked by approximate importance
For each concept, available instruments are also
described (researchers prioritize)
Framework: Eight Therapy Measurement
Domains, with examples of key concepts
Research Theme
A. General/
Pantheoretical
B. TreatmentSpecific
I. Therapy
Outcome
e.g., general problem e.g., theory-based
severity
dysfunctional
processes
II. Therapy
Process
III. Client/Therapist Background
e.g., general problem e.g., therapist
severity
techniques
e.g., demographics
e.g., preference for
type of therapy
IV. Training
Outcome
e.g., productive vs.
e.g., therapist skill
unproductive practice development
pattern
Example: General Therapy
Outcome Domain

Key concepts in a possible recommended
priority order: (“Star”)

(1) General problem severity (quantitative)






Give every 2 sessions to reduce data loss from drop-out
(2) Interpersonal/relational functioning
(3) Qualitative perceptions of change
(4) Individualized problems/goals
(5) Health care utilization/costs
(6) Quality of life/life satisfaction/well-being
Common General Symptom Severity Instruments
In strumen t (length )
Refe rence
Symptom Checklist 90-Revised (SCL-90R) (90 items)
Clinical Outcomes in
Rout ine Evaluation
Outcome Measure
(CORE-OM; 34
items)
Outcome
Quest ionnaire (OQ45) (45 items)
Treatment Outcome
Package (TOP) Adult
Clinical Scales v4.0
(58 items)
S horter
Forms
(length )
Scale Basi s,
Poin ts &
Ti me Frame
C ost
Non Engl ish
Translati on s
10 (incl.
Italian)
Derogat is
Brief
et al., 1976 Symptom
Inventory (53
items)
Evans et
CORE-SF
al., 2002
(18 items)
Dist ress
5 point s
Past week
$2 each
(scoring
ext ra)
Frequency
5 point s
Past week
Free
Italian,
Slovak,
Norwegian
Lambert et
al., 1996
OQ-30 (30
items); OQ10 (10 items)
Frequency
5 point s
Past week
German,
Dutch,
Spanish,
Italian
Kraus et
al., 2005
TOP 37 (37
items)
Frequency
6 point s
Past 2 weeks
One-t ime
licensing
fee (e.g.,
$30 for a
student)
Free
Spanish
Framework: Eight Therapy Measurement
Domains, with examples of key concepts
Research Theme
A. General/
Pantheoretical
B. TreatmentSpecific
I. Therapy
Outcome
e.g., general problem e.g., theory-based
severity
dysfunctional
processes
II. Therapy
Process
III. Client/Therapist Background
e.g., therapeutic
alliance
e.g., demographics
IV. Training
Outcome
e.g., productive vs.
e.g., therapist skill
unproductive practice development
pattern
e.g., therapist
techniques
e.g., preference for
type of therapy
Example: General Therapy
Process Domain

Key concepts in possible recommended priority
order:




(1) Therapeutic alliance
(2) Therapist and client response modes
(3) Perceived helpful aspects of therapy
(4) Perceived session effectiveness
Common Therapeutic Alliance Instruments
In strumen t
(items)
Refe rence
S ubscales
In formant
Non -Engl ish
Trans lation s
Horvath &
Greenberg,
1989;
Hatcher,
2005a
Scale B asi s,
Poin ts &
Ti me Frame
Frequency
7 point s
(revised short
form: 5
points)
W orking
Alliance
Inventory
(W AI) (36
items; 12 item
short form)
California
Psychotherapy
Alliance Scale
(CALPAS; 24
items)
Bond
Task
agreement
Goal
agreement
Client
Therapist
Observer
Dutch
Danish
French
Italian
Gaston &
Marmar,
1994
Agreement Disagreement
6 point s
Pat ient
W orking
Capacity
Pat ient
Commit ment
Therapist
Understanding
and
Involvement
W orking
St rategy
Consensus
--
Client
Therapist
French
Portuguese
Italian
Penn Helping
Alliance
Quest ionnaire-II
(Haq-II; 19
items)
Luborsky
Agreement
et al., 1996 7 point s
Client
Therapist
German
French
Norwegian
Dutch
(earlier
version)
Different Levels of Research
Protocol are Possible
I. Minimum Protocol
 II. Systematic Case Study Protocol
 III. Maximum Protocol
Other Protocols:
 IV. General Training Protocols
 V. Specific Research Protocols

I. A Recommended Minimum
Protocol: Applications




Easy to use: Limited to one measure from
each of the first three research domains
Can use with own clients
Provides basic treatment monitoring for
individuals & agencies
Other versions are possible (e.g., different
outcome or process measures)
I. A Recommended Minimum
Protocol: Elements

(1) General therapy outcome instrument



(2) General therapy process


Client problem severity
Give at odd-numbered sessions (short form)
Therapeutic alliance (use short from)
(3) Client/therapist background measure




Standard practice:
Client/ therapist demographics
Client diagnosis, presenting problems
Type of therapy
II. Systematic Case Study
Protocol: Applications
Use for student case study requirements
 Meets emerging standards for systematic single
case research


New online journal: Pragmatic Case Studies in
Psychotherapy (Rutgers University, Editor: Fishman)
II. Systematic Case Study
Protocol: Elements

A. Therapy Outcome:
 (1) Weekly/biweekly outcome measure
+ (2) At least one other quantitative outcome measure
+ (3) Qualitative outcome assessment (e.g., post-therapy interview)

B. Therapy Process
 (1) Therapeutic alliance
+ (2) Detailed record of therapy (process notes and/or recordings)
+ (3) Qualitative perception of helpful aspects (post-session and/or posttherapy)

C. Client/therapist background
 Client/ therapist demographics; client diagnosis, presenting problem;
type of therapy
II. Systematic Case Study
Protocol: Research Questions



(1) Did the client change substantially
over the course of therapy?
(2) If the client changed, did therapy
make a substantial contribution?
(3) What brought about the client’s
changes?
II. Systematic Case Study
Protocol: Emerging Evidence
Standards



(1) Rich case record, including both quantitative
& qualitative data
(2) Replication/convergence across methods
(3) Critical examination of alternative views
(e.g., Hermeneutic Single Case Efficacy Design,
Elliott, 2002):



Non-change explanations (e.g., measurement error)
Non-therapy explanations (e.g., extra-therapy events)
(4) Narrative coherence


Narrative model of predisposing and process factors
Use for generalizing to other cases
III. Maximum Protocol



Include measures of at least one concept in
each of the eight domains
Appropriate for research centers (e.g.,
Center for the Study of Experiential
Psychotherapy)
Also consortia of cooperating centers:

Each center measure some variables
IV. General Training Research
Protocols: Issues


Outcomes of therapy training not well understood
Difficulties:



Lack of agreed-upon measures of therapist functioning
and skill
Must measure therapist change longitudinally over
several years of training
Possible applications:


Use research to improve training
Meet requirements of accrediting and funding agencies
IV. General Training Research
Protocols: Promising Concepts


General therapist facilitative interpersonal
skills (e.g., coping with common difficulties)
Quality of therapist professional involvement
and growth (e.g., Orlinsky; Collaborative Research
Network [CRN])

Qualitative perceptions of effects and
important aspects of training (e.g., qualitative
interviews)

Change in therapist self concept (e.g., Scilligo,
SASB Introject scales)
V. Specific Protocols



= Star rays
Applications: For specific theoretical
approaches, client populations, or language
groups
Requires working committee for each group



Identify relevant therapy outcomes, processes,
background variables (or training outcomes)
Do protocol and measure development research
Establish virtual communities for exchanging
V. Specific Protocol Example:


Person-Centered and Experiential
Psychotherapy International Research
Group (PCEP-IRG)
Current core members:




University of Toledo (Elliott & team)
Ohio University (Anderson & team)
Katholieke Universiteit Leuven (Leijssen & team)
Universities of Strathclyde & Abertay, Scotland
(McLeod, Cooper)
V. PCEP-IRG Outcome Protocol:
Promising Developments


Center for the Study of Experiential Therapy
Research Protocol (CSEP- 2):
Self--determined problems/goals:


Self-concept (content & coherence)



Qualitative Self-Description interview
Tennessee Self-Concept Scale 2 (long, short forms)
Experiential processing:


Personal Questionnaire (PQ-10)
Toronto Alexithymia Scale (TAS-20)
Need positive mental health measures, selfcoherence, etc.
V. Specific Training Research
Protocols

Some Possible Types of Specific Training
Outcomes:
Treatment-specific intervention skill
 Case formulation skill
 Therapist personal development (e.g.,
maturity, identification with orientation,
values)

Promising New Therapy Research
Methods Make this Work Possible





Systematic qualitative research methods
Interpretive single case designs (Fishman,
Elliott)
Using early outcome to identify & repair
problems (Lambert: Signal alarm methods)
New, powerful psychometric methods
(Rasch analysis/Item Response Theory)
Virtual communities (Community Zero)
Invitation to Dialogue - 1
1) Provide comments and suggestions on the
framework & concepts presented here:
[email protected]
2) Form or join online discussion groups or virtual
communities



Closed sites; must apply for membership
General info: www.communityzero.com/ipeppt
Example: www.communityzero.com/pcepirp
3) Begin implementing the minimum protocol design
with your own clients and in your own training
setting.
Invitation to Dialogue - 2
4) Convert traditional case presentation training
requirements into systematic case study exercises
5) Help with translations of key research instruments
6) Contribute to psychometric research:


Improve existing instruments
Equate different instruments for same concepts
7) Collaborate with groups with similar interests to
generate data for pooling.