Transcript Slide 1

TeamSTEPPS
Measurement and Evaluation
of TeamSTEPPS
Mod 1 05.2 Page 1
TEAMSTEPPS 05.2
Measurement and Evaluation
of TeamSTEPPS
TeamSTEPPS National Conference
June 22, 2012
TeamSTEPPS
Session Moderator
 Andrea Amodeo, MS, Research Associate, IMPAQ
International
Mod 1 05.2 Page 3
TEAMSTEPPS 05.2
TeamSTEPPS
Purpose
 Provide Information on How to Measure and
Evaluate TeamSTEPPS’ Success
 Provide Best Practices
 Describe an Example
Mod 1 05.2 Page 4
TEAMSTEPPS 05.2
TeamSTEPPS
Agenda
 Introduction
 Why Measure and What to Measure: Perspectives
from the C-Suite
 Ten Considerations for Measurement
 Developing a Measure of Trauma Team
Performance: An Example
 Panel Q&A
Mod 1 05.2 Page 5
TEAMSTEPPS 05.2
TeamSTEPPS
Panel Introductions
 Anthony Slonim, MD, DrPH, Barnabas Health
 Eduardo Salas, PhD, University of Central Florida
 David P. Baker, PhD, IMPAQ International
Mod 1 05.2 Page 6
TEAMSTEPPS 05.2
TeamSTEPPS
Why Measure and What to
Measure:
Perspectives from the C-Suite
Anthony Slonim, MD, DrPH
Executive Vice President and Chief Medical Officer,
Barnabas Health
Professor, Medicine, Pediatrics,
Preventive Medicine and Community Health
University of Medicine and Dentistry of New Jersey
New Jersey Medical School
Mod 1 05.2 Page 7
TEAMSTEPPS 05.2
TeamSTEPPS
Acknowledgments
 No Conflicts of Interest
 AHRQ Funding and Contracts:

Contract # HHSA290200600019i Task Order #12 Subcontract:
SAMIKE, LLC: “Proactive Risk Assessment in Ambulatory
Surgery Centers” Anthony D. Slonim, MD, DrPH Principal
Investigator.”
 Objectives

Participants will understand the importance of
measuring teamwork success

Participants will understand what measures are
T
STEPPS 05.2
important for leadership
Mod 1 05.2 Page 8
EAM
TeamSTEPPS
Why Measure?
 To know if your TeamSTEPPS implementation




worked
 Step 5 of Implementation Planning
Assist in modifying implementation plan
Planning future implementations
Information for leadership
Producing a scholarly product
Mod 1 05.2 Page 9
TEAMSTEPPS 05.2
TeamSTEPPS
TeamSTEPPS Phases
Mod 1 05.2 Page 10
TEAMSTEPPS 05.2
TeamSTEPPS
TeamSTEPPS Phases
Mod 1 05.2 Page 11
TEAMSTEPPS 05.2
TeamSTEPPS
TeamSTEPPS Phases
Outcomes
Mod 1 05.2 Page 12
TEAMSTEPPS 05.2
TeamSTEPPS
Teamwork
Climate
Provider
to
Patient
Provider
Macrosystem
Outcomes
To
Provider
Microsystem
Mod 1 05.2 Page 13
TEAMSTEPPS 05.2
Patient
TeamSTEPPS
Teamwork
Provider
to
Patient
Climate
Provider
Macrosystem
To
Outcomes
Provider
Microsystem
Mod 1 05.2 Page 14
TEAMSTEPPS 05.2
Patient
TeamSTEPPS
Macrosystem
Safe
Equitable
Microsystem
Safe
Patient
Centered
Safe, Equitable,
Pt
Centered
Equitable
Effective
Effective,
Timely,
Efficient
Patient
Centered
Timely
Efficient
Mod 1 05.2 Page 15
TEAMSTEPPS 05.2
TeamSTEPPS
How do Hospitals Work?
H
CEO/Management Team
Mod 1 05.2 Page 16
TEAMSTEPPS 05.2
TeamSTEPPS
Teamwork
Provider
to
Patient
Climate
Provider
Macrosystem
Outcomes
To
Provider
Microsystem
Mod 1 05.2 Page 17
TEAMSTEPPS 05.2
Patient
TeamSTEPPS
The Microsystem Team
Patient
Teamwork
Family
Mod 1 05.2 Page 18
Providers
TEAMSTEPPS 05.2
TeamSTEPPS
Teamwork
Provider
to
Patient
Climate
Provider
Macrosystem
Outcomes
To
Provider
Microsystem
Mod 1 05.2 Page 19
TEAMSTEPPS 05.2
Patient
Patient
TeamSTEPPS
Structure
People
Access
Process
Interactions between:
Outcomes
Providers and Patients
Vitality
Buildings
Providers and Providers
Economic
Technology
Organization
•Trained Providers
•Primary Care Services
•EMS
•Emergency Departments
•ICUs
•Specialized Equipment
•Regionalized Care
•Financing
Mod 1 05.2 Page 20
Providers and Technology
•Evidence Based Practices
•‘Bundles’ of Care
•Specialist Availability
•Family Centered Care
•Trainees
•Teamwork
•Multidiscplinary Care
Quality
•Mortality
•Cost
•Length of Stay
•Infections
•Complication Rates
•Quality of Life
Slonim AD, Marcin JP, and Pollack MM. Outcomes in Pediatric Critical Care
Medicine: Implications for Health Services Research and Patient Care. In:
Fuhrman B and Zimmerman J. Pediatric Critical Care 4th Edition Mosby 2011.
TEAMSTEPPS 05.2
TeamSTEPPS
Clinical Processes
Provider to Provider:
 Policies and




Mod 1 05.2 Page 21
Provider to Patient:
 Safe
Procedures
Core Measures
Bundle Adherence
Family Centered
Teamwork
 Availability
 Consistency
 Knowledgeable
TEAMSTEPPS 05.2
 Effective
 Outcome Based


Clinical
Experiential
Bedside Care
TeamSTEPPS
Outreach
Sales,
Marketing
Welcome, Register, Risk Assess, Triage
Relationships
and
Transport
Needed
Services
Provide Feedback and Monitor Service
Care Team
Introductions
And Follow-up
Assess,
Plan
And
Initiate
Care
Direct
Care
Delivery
Technology
Care
Delivery
Treatment
Plan
And
Care
Delivery
Deliver
Care
Clinical
Support
Services
Evaluate Care, Outcomes and Business
Slonim AD, Marcin JP, and Pollack MM. Outcomes in Pediatric Critical Care Medicine: Implications for Health
Services Research and Patient Care. In: Fuhrman B and Zimmerman J. Pediatric Critical Care 4th Edition Mosby
2011.
Mod 1 05.2 Page 22
TEAMSTEPPS 05.2
TeamSTEPPS
Teamwork
Provider
to
Patient
Climate
Provider
Macrosystem
To
Outcomes
Provider
Microsystem
Mod 1 05.2 Page 23
TEAMSTEPPS 05.2
Patient
TeamSTEPPS
Culture vs. Climate
Climate:
 Organizational Structure

Connectiveness
 Historical/Environmental
Forces
 Vision/Strategy
 Standards


Accountability
Behavior
 Communication
 Rewards
 Trust
Mod 1 05.2 Page 24





Culture:
Values
Beliefs
Myths
Norms
Traditions
Kennedy Group Executive Strategies:
Consulting at the Kennedy Group.com
TEAMSTEPPS 05.2
TeamSTEPPS
Teamwork
Provider
to
Patient
Climate
Provider
Macrosystem
Outcomes
To
Provider
Microsystem
Mod 1 05.2 Page 25
TEAMSTEPPS 05.2
Patient
TeamSTEPPS
What Leadership Cares About:
An Integrated Quality Program: Content
Clinical Quality:
•Continual Survey Readiness
•Core Measures + Pain
• Patient Centered:
•HCAHPS Survey Scores
•Nursing and Doctor Quality
•Likelihood to Recommend
•Readmissions
•Wait times and Pain/empathy
•Standardized Ratios
• Employee Centered:
•LOS
•Engaged in Mission
•Mortality
•Accountable
•PSIs/PQIs/PPCs
Mod 1 05.2 Page 26
Patient Experience:
•Delivering on Excellence
TEAMSTEPPS 05.2
TeamSTEPPS
Practice Evaluations and Learning
Activities
 Performance achievement
 High-recognize and promote
 Moderate-target improvement opportunities
 Low






Mod 1 05.2 Page 27
Graduated Counseling
Education
Supervision
Assurance of competency
Behavior modification
Other areas for improvement
TEAMSTEPPS 05.2
TeamSTEPPS
Teamwork
Provider
to
Patient
Climate
Provider
Macrosystem
Outcomes
To
Provider
Microsystem
Mod 1 05.2 Page 28
TEAMSTEPPS 05.2
Patient
TeamSTEPPS
The Patient’s Responsibility
 Open, Honest
 Follows Through
Therapy
 Recommendations
 Rules of engagement
 Asks Questions, Demands Answers

Mod 1 05.2 Page 29
TEAMSTEPPS 05.2
TeamSTEPPS
Mod 1 05.2 Page 30
TEAMSTEPPS 05.2
TeamSTEPPS
Designing a Team Performance
Measurement System:
Ten Considerations
Eduardo Salas, PhD
Department of Psychology &
Institute of Simulation & Training
University of Central Florida
[email protected]
Mod 1 05.2 Page 31
TEAMSTEPPS 05.2
TeamSTEPPS
To Begin…
 Measurement is not “Sexy,
Flashy”…Yet, of the utmost importance!
 No “Silver Bullet”…

No Perfect Tools!
 All need to be adapted, refined,
expanded for different purposes!
Mod 1 05.2 Page 32
TEAMSTEPPS 05.2
TeamSTEPPS
1. Consider Level of Analysis
 Individual
MD
 Nurse
 Technician
 Team
 Emergency Department Team
 Radiology Team
 Multi-team – Team Structure
 Core Care Team
 Contingency Team (e.g., RRT)
 Administrative Team

Mod 1 05.2 Page 33
TEAMSTEPPS 05.2
TeamSTEPPS
2. Clarify the Purpose
 Diagnose root causes of performance
deficiencies
 Identify specific weaknesses
 Provide feedback

Relay information regarding strengths and
weaknesses as a remediation plan
 Assessment

Mod 1 05.2 Page 34
Evaluate the level of proficiency or readiness
TEAMSTEPPS 05.2
TeamSTEPPS
3. Decide What to Measure:
Outcomes vs. Processes
Mod 1 05.2 Page 35
TEAMSTEPPS 05.2
TeamSTEPPS
4. Select a Measure: Checklist
 Items/actions use dichotomous responses
•
YES/NO
•
RIGHT/WRONG
•
PERFORMED/NOT PERFORMED
Mod 1 05.2 Page 36
TEAMSTEPPS 05.2
TeamSTEPPS
4. Select a Measure: Frequency
Count
 # of times a behavior, action, or error occurs



Mod 1 05.2 Page 37
Better for measuring acts of commission vs. acts
of omission
Useful when purpose = know how often a
specific action is taken or task is performed
Can be recorded during a critical event in an
exercise or across the entire scenario
TEAMSTEPPS 05.2
TeamSTEPPS
4. Select a Measure: Graphic
Rating Scale
 Numeric or descriptive judgment of how well a
task was performed
1. The Team Leader assigned roles to the Trauma Team
Very
Effective
Ineffective
1
2
3
4
5
6
2. The PGY2 used check back to confirm orders.
Very
Effective
Ineffective
1
Mod 1 05.2 Page 38
2
3
4
TEAMSTEPPS 05.2
5
6
TeamSTEPPS
4. Select a Measure: Anchored
Rating Scale
Mod 1 05.2 Page 39
TEAMSTEPPS 05.2
TeamSTEPPS
5. Decide the Timing; When?
Mod 1 05.2 Page 40
TEAMSTEPPS 05.2
TeamSTEPPS
6. Consider Fidelity of Setting;
Where?
Mod 1 05.2 Page 41
TEAMSTEPPS 05.2
TeamSTEPPS
7. Train Observers; Who?
 Choosing observers
Clinical + Teamwork competence
 Training and supporting raters
 Develop and maintain high inter-rater reliability
 Rater training
 Scoring guides
 Developing coaches and facilitators
 How is data going to be used to improve
performance?
 Facilitation skills
 Debrief assessment

Mod 1 05.2 Page 42
TEAMSTEPPS 05.2
TeamSTEPPS
8. Calibrate the Measurement
System: Test It!
Mod 1 05.2 Page 43
TEAMSTEPPS 05.2
TeamSTEPPS
9. Dilemma – Generic vs. Specific
Tools
Mod 1 05.2 Page 44
TEAMSTEPPS 05.2
TeamSTEPPS
10. It is Paramount for Debriefing!
 To maximize learning from experience
 Practice alone isn’t good enough
 Structure practice with diagnostic feedback is required
 To ensure that the ‘right’ lessons are learned
 Everyone walks away with the same lessons learned
 The team’s interpretation of what happened and why it
happened is cross-checked with standards
 To promote self-reflection and continuous
learning


Mod 1 05.2 Page 45
Develop the team’s self-correction skills
‘Calibrate’ team members to rating their own
performance
TEAMSTEPPS 05.2
TeamSTEPPS
Remember the Issues to Consider…
 Why…Always keep purpose in mind!
 “In the end, are the questions this measurement tool
could answer what I really want to know?”
 What…What content needs to be captured by the
measurement system?
 Where…Where will team be assessed?
 Training room vs. in-situ
 When….when is it best to measure each
competency in the scenario?
 Who…Who will be using this measure? What
training will they have?
 How… What scale is best?
Mod 1 05.2 Page 46
TEAMSTEPPS 05.2
TeamSTEPPS
Developing a Measure of Trauma
Team Performance:
An Example
David P. Baker, PhD
IMPAQ International
Mod 1 05.2 Page 47
TEAMSTEPPS 05.2
TeamSTEPPS
Acknowledgements
 No conflicts of interest
 Part of Dr. Jeannette Capella’s Surgical Education
and Research Fellowship
 Research Team




Mod 1 05.2 Page 48
Jeannette Capella, MD (PI)
Andi Wright, RN
Ellen Harvey, RN
Sonya Ranson, PhD
TEAMSTEPPS 05.2
TeamSTEPPS
Purpose
 To develop and test a new tool for observing and
measuring team performance during trauma
resuscitation
 Trauma Team Performance Observation Tool
(TPOT)
 To conduct an investigation of the impact of
TeamSTEPPS on:
 Trauma team performance
 Patient outcomes
Mod 1 05.2 Page 49
TEAMSTEPPS 05.2
TeamSTEPPS
Background
 Within the OR and trauma room, there has been a
growing body of evidence validating the
importance of teamwork
 Christian et al. (2006)


A prospective study of ten general surgery cases
Problems in communication, managing workload, and
prioritizing competing tasks within the surgical team
were found in all ten cases
 Few team performance evaluation tools,
particularly in the areas of surgery and trauma
Mod 1 05.2 Page 50
TEAMSTEPPS 05.2
TeamSTEPPS
Methods
 Conducted interviews
31 trauma team members (e.g., physicians,
nurses, residents)
 Two different organizations
 Goals of the interviews
 To identify the steps in trauma resuscitation
 To identify technical and team requirements that
comprise each step
 Similar to a mini FMEA

Mod 1 05.2 Page 51
TEAMSTEPPS 05.2
Phase
Technical
Teamwork
Transport
1. EMS/rescue team or trauma team
brings patient to resuscitation/trauma
area
2. EMS/rescue team and/or trauma
team continues ABCs: assesses
patient airway, breathing, circulation,
disability, and exposure/environment
3. EMS/rescue team conducts verbal
handoff of patient information to
trauma team
(Communication)
1. Team members are
quiet while EMS/rescue
team gives report
1. Perform ABCs—this should be
completed in 60-90 seconds:
A: Airway (secure airway; identify
problems; initiate timely
intervention)
B: Breathing (assess lung sounds;
identify problems; initiate timely
intervention)
[….and much more….]
(Leadership)
1. Team leader continually
communicates and
advocates the plan of
care
(Situation Monitoring)
1. Team members know
their role and
responsibilities
2. Team member(s)
prepare(s) patient
and/or equipment
TeamSTEPPS
Results
Primary
Survey
Mod 1 05.2 Page 52
TEAMSTEPPS 05.2
TeamSTEPPS
Methods
 An initial pool of items was developed for the
TPOT through an extensive item writing process
 Items were linked to the following four team
constructs:
 Leadership
 Situation monitoring
 Mutual support
 Communication
Mod 1 05.2 Page 53
TEAMSTEPPS 05.2
Phase
Technical
Teamwork
Transport
1. EMS/rescue team or trauma team
brings patient to resuscitation/trauma
area
2. EMS/rescue team and/or trauma
team continues ABCs: assesses
patient airway, breathing, circulation,
disability, and exposure/environment
3. EMS/rescue team conducts verbal
handoff of patient information to
trauma team
(Communication)
1. Team members are
quiet while EMS/rescue
team gives report
1. Perform ABCs—this should be
completed in 60-90 seconds:
A: Airway (secure airway; identify
problems; initiate timely
intervention)
B: Breathing (assess lung sounds;
identify problems; initiate timely
intervention)
[….and much more….]
(Leadership)
1. Team leader continually
communicates and
advocates the plan of
care
(Situation Monitoring)
1. Team members know
their role and
responsibilities
2. Team member(s)
prepare(s) patient
and/or equipment
TeamSTEPPS
Results
Primary
Survey
Mod 1 05.2 Page 54
TEAMSTEPPS 05.2
TeamSTEPPS
Leadership – The team leader …..
1. Conducts a brief prior to patient arrival (e.g., identifies
2.
3.
4.
5.
self, assigns members roles and responsibilities,
discusses initial plan based on current information,
anticipates interventions [e.g., chest tube, OR])
Continually renders plan of care to team
Feedback provided to team members is constructive
Ensures task prioritization (e.g., important tasks
performed first, ABC’s and survey sequence are being
completed)
Asks non-response team members to leave when they
are distracting
Mod 1 05.2 Page 55
TEAMSTEPPS 05.2
TeamSTEPPS
Situation Monitoring – Team members ….
1. Prepare equipment before patient arrival (e.g., set up IV,
2.
3.
4.
5.
6.
Mod 1 05.2 Page 56
ultrasound machine, suction)
Work quickly and efficiently
Conduct tasks in right order
Are not distracted by major injuries
Ensure that NEW team members perform expected role
and responsibilities
Adapt quickly and efficiently to deterioration of patient’s
condition (e.g., decreased O2 sats, decreased blood
pressure, decreased mental status)
TEAMSTEPPS 05.2
TeamSTEPPS
Mutual Support – Team members ….
1. Feedback provided to other team members is
constructive
2. Assist when moving patient to next unit (e.g., CT
scanner, OR, ICU)
3. Provide assistance when needed/Complete other
team members’ tasks
4. Identify/Call out when patient safety issue is
suspected
Mod 1 05.2 Page 57
TEAMSTEPPS 05.2
TeamSTEPPS
Communication – Team members ….
1. Remain quiet while team gives report
2. Request additional information from EMS (e.g.,
3.
4.
5.
6.
Mod 1 05.2 Page 58
medications given, vital signs, mechanism of injury)
Use call-outs to share important patient information (i.e.,
Team leader “Airway status?” Airway doc responds
“Airway clear!”)
Use check-backs to verify important information is
exchanged (i.e., Doctor “Give 25 mg Benadryl IV.” Nurse
“25 mg Benadryl IV” to confirm. Doctor “That’s correct”)
Use clear and concise language
Request information from others when it’s not readily
shared
TEAMSTEPPS 05.2
TeamSTEPPS
Scoring
1
2
Very
Poor
poor
Very
Poorly
poorly
done
done
Should
Should
have
have been
been done more
done
often
but was
not
Mod 1 05.2 Page 59
Rating Scale
3
4
Average
Good
Acceptable
performance
Could have
been done
more often/
consistently
but is
acceptable as
is
NA
Not
applicable
Good
Perfect
Did not
performance Performance
need to
be done
Done most of Done at all
Was not
the time
times
done and
appropriately
did not
need to
be done
TEAMSTEPPS 05.2
5
Excellent
TeamSTEPPS
Observer Training
 Five staff (two trauma nurses and three trauma
registrars) were trained to use the TPOT
 Training involved four steps
1.
2.
3.
4.
Mod 1 05.2 Page 60
Reviewing and discussing the TPOT
Practice and feedback rating the videotapes
Revising the TPOT items for clarity, as needed
Independent evaluation of the videotapes
TEAMSTEPPS 05.2
TeamSTEPPS
Rater Agreement
Scenario
Team
Mean
ICC
Inter-Rater
Agreement
1
1
2.67
.44
65%
2
1
1.22
.64
82%
1
2
2.98
.47
69%
2
2
1.21
.58
86%
Mod 1 05.2 Page 61
TEAMSTEPPS 05.2
TeamSTEPPS
Implementation
 Part of a larger study to assess the impact of
TeamSTEPPS
 Trained raters
 Observed and rated 33 live trauma resuscitations
in the trauma bay over a 3-month period (PreTraining)
 Observed and rated 40 live trauma resuscitations
in the trauma bay over a 3-month period (PostTraining)
Mod 1 05.2 Page 62
TEAMSTEPPS 05.2
TeamSTEPPS
Alphas and Inter-correlations
Team Skills
LDR
SM
MS
COM
Leadership
Situation
Monitoring
.92
Mutual Support
.75
.75
Communication
.85
.82
.70
Alpha
.53
.57
.64
Mod 1 05.2 Page 63
TEAMSTEPPS 05.2
.63
Mean
Stdev
2.90
.68
3.29
.62
3.88
.47
2.92
.58
TeamSTEPPS
 Design


Impact Study
Quasi-experiment
Pre-test/Post-test, no control group design
 Interventions


Training – Didactic and Simulation
2-hour TeamSTEPPS Essentials; 2 hours
simulation (skills practice and feedback)




Mod 1 05.2 Page 64
Nurses and Doctors separately trained
Trauma Room Roles and Responsibilities Policy
Briefing, STEP, CUS, Call-Outs and Check-Backs
One-day Nursing Crash Course
TEAMSTEPPS 05.2
TeamSTEPPS
Measures (Kirkpatrick)
 Level I Reactions

End of Training
 Level II Learning – Pre/Post Test

Knowledge, Attitudes, Skills
 Level III Transfer – Observed Teamwork (TPOT)

3 months pre and post-training
 Level IV Outcomes


Mod 1 05.2 Page 65
Clinical Outcomes (ICU LOS, Hospital LOS, Complication
Rate, Mortality)
Clinical Process (Time to CT SCAN, Surgery, Intubation)
TEAMSTEPPS 05.2
TeamSTEPPS
Expected Findings
 Reactions - Positive
 Learning



Knowledge

No change – staff know what to do
Attitudes

Positive – Believe in teamwork or social desirability
Skills

Significant improvement in simulator
 Transfer

Not sure about trauma bay – many environmental factors
 Outcomes

Mod 1 05.2 Page 66
Unlikely due to base rate issues
TEAMSTEPPS 05.2
TeamSTEPPS
Descriptive Data
Mod 1 05.2 Page 67
Pre-Training
Post-Training
Observed
33
40
Trauma
176
263
ISS
M=13.97,
SD=11.85
M=11.63;
SD=11.04
TEAMSTEPPS 05.2
TeamSTEPPS
Level III Transfer
Pre-training
N=33
Post-training
N=40
p value
Leadership
2.87
3.46
0.003
Situation monitoring
3.30
3.91
0.009
Mutual support
3.40
3.96
0.004
Communication
2.90
3.46
0.001
Total
3.12
3.70
<0.001
Note: Pre-training and post-training reflect observations in trauma bay
by 4 trained raters.
Mod 1 05.2 Page 68
TEAMSTEPPS 05.2
TeamSTEPPS
Mod 1 05.2 Page 69
TEAMSTEPPS 05.2
TeamSTEPPS
Challenges
 Measurement
 How best to train observers?
 Moderate reliability
 Capella & Baker, ACOS Grant to study different
training strategies

Mod 1 05.2 Page 70
Available tools
 Had to develop tools, few available, few validated
TEAMSTEPPS 05.2
TeamSTEPPS
Thank You!
For more information, please contact our team at:
[email protected]
Mod 1 05.2 Page 71
TEAMSTEPPS 05.2