Transcript An Introduction to Knowledge Translation Theory and Practice

Characterization and analysis of
guideline implementation tools
(GItools)
Anna R Gagliardi, PhD
Scientist, Toronto General Research Institute
Associate Professor, University of Toronto (Department of Surgery; Institute of
Health Policy, Management & Evaluation; and Institute of Medical Science)
CIHR New Investigator in Knowledge Translation (2008-2013)
Chair, Implementation Working Group, Guidelines International Network
funding provided by the Canadian Institutes of Health Research
1
Outline
• Why are guidelines relevant?
• If they are relevant, why is implementation challenging?
• What is guideline implementability?
• How do guideline implementation tools (GItools)
enhance guideline implementability?
• GItool Framework development
• Analysis of GItools with the GItool Framework
• Summary of findings
• Next steps
2
Reflection
• Should all guidelines offer implementation tools?
• How should implementation tools be developed (rigor)?
• How do you decide what type of tools are needed?
• With respect to timing of guideline development, when
should tools be developed?
3
Evidence not translated to practice
• Guidelines are “one of the foundations for efforts to improve health
care” (Shekelle 2012)
• United States – 55% of patients received care based on 439
recommendations for 30 conditions (McGlynn 2004)
• United Kingdom – care improved for 3 of 12 recommendations
among 58 trusts (Sheldon 2004)
• MEDLINE 2004+ – guideline adherence 15,573 hits
4
Implementation is challenging
Intervention
Effect
Optimization
Educational
meetings
small
interactive, sequential, participants involved in planning,
reflection in/on practice, commitment to change
Decision
support tools
moderate
training, interface, workflow, integration with care and
care delivery mechanisms
Public
reporting
small
pay-for-performance; report content, format, delivery
Audit and
feedback
small to
moderate
baseline adherence low, data individualized, delivery
interactive and periodic, report content/format
Opinion
leaders
small
use multiple methods to identify and train OL’s, and
offer multiple opportunities for interaction with targets
Patient
strategies
moderate
appropriate supports for specific contexts
5
Extrinsic determinants of use
Innovation Attributes
advantage, trialability, compatibility,
uncertainty, complexity
Implementation Strategies
educational, social,
organizational, incentives,
embedded
Use of
Innovation
Outcomes
Individual Attributes
Organizational Attributes
professional role, years in
practice, networking
absorptive capacity, leadership,
teamwork, system readiness,
support for quality improvement
6
Intrinsic determinants of use
• Sheer volume is overwhelming to health professionals
• Difficult to harmonize recommendations across different guidelines for
patients with multiple conditions
• Underlying evidence lacking, conflicting or of poor quality
• Inconsistencies in recommendations developed by different agencies
for same condition or procedure
• Lack of methodologic rigour – quality of content, format
• Expert consensus may introduce bias, conflict of interest
• Limited applicability to individual patients
• Loss of professional autonomy / medico-legal implications
• Lengthy, complicated documents
• Implementation often a user responsibility
• Not readily available at point-of-care
7
Guideline implementability
• “Characteristics of guidelines that may enhance their implementation by
users” (Shiffman 2005)
• Appraisal of Guidelines for Research & Evaluation Instrument (AGREE)
www.agreetrust.org
• Grading of Recommendations Assessment, Development and
Evaluation (GRADE)
www.gradeworkinggroup.org
• Guideline Implementability Appraisal Instrument (GLIA)
nutmeg.med.yale.edu/glia/login.htm
8
Users need implementation support
• Cognitive science theory – guidelines present complex information and
are difficult to use (Patel 2001)
• Systematic review – adoption and adherence low even when
awareness and agreement high (Mickan 2011)
• Guideline users – interviews found they were frustrated and at a loss to
about how to implement guidelines (McKillop 2012)
• Empirical research – guideline use was greater for those featuring
implementation instructions or tools (Cochrane 2007; Dobbins 2009; Shekelle 2000)
• JAMA editorial – guideline developers should provide clinicians with
tools that support implementation (Pronovost 2013)
9
Monitor
Knowledge
Use
Select, Tailor,
Implement
Interventions
Assess
Barriers to
Knowledge Use
KNOWLEDGE CREATION
Knowledge
Inquiry
Synthesis
Adapt
Knowledge
to Local Context
Evaluate
Outcomes
Products
Tools
Sustain
Knowledge
Use
Identify Problem
Identify, Review,
Select Knowledge
10
Guideline Implementation tools (GItools)
Canadian Stroke Care Recommendations
Domain
Example
Description
Resource
planning
Unit Care
Implementation
Guide
Print and electronic booklet including definitions and
key components of stroke units, steps for setting up
a stroke unit, and staffing models
Implementation
Implementation
Toolkit
Educational modules, self-directed learning
resources including case studies
Workshop Toolkits
Slide presentations with speaker notes, handouts,
and templates for organizing/delivering workshops
Patient
engagement
Patient Guide to
Best Practice
Recommendations
Signs/symptoms, how to react, expectations while I
hospital, rehabilitation and prevention
Evaluation
Performance
Measurement
Manual
Performance measurement framework, benchmarks
for stroke care, instructions on how to select,
implement and monitor performance measures
11
GItool development support needed
• Interviews with 30 guideline developers from 7 countries:
– Demand among constituents for implementation tools
– Feasible approach so could be widely adopted
– Few had developed GItools, requested guidance
(Gagliardi 2012A)
• Review of guideline development instructional manuals revealed
they lack GItool development guidance (Gagliardi 2012B, Schunemann 2014)
12
Study Overview
Objectives
• To generate a framework of ideal GItool features
• Use the framework to analyse a sample of GItools
Implications
• Baseline assessment of the status of the field
• Framework could be used to:
• Assess, and adapt/adopt or endorse existing GItools
• Develop new GItools
13
Part 1 – candidate GItool features
• Cross-sectional Internet survey of Guidelines International Network
members (107 individuals, 86 organizations, 45 countries)
• G-I-N secretariat distributed email invitation in July 2012 with reminders
at two and four weeks
• Candidate features based on items considered essential for evaluation
or measurement instruments
• Rated desirability of each for assessing/developing GItools on 7-point
Likert-type scale with free text option
• 96 respondents from 12 countries
• No new items generated
14
Results: rating of GItool features
GItool Feature
Agree or
Strongly
Agree
1. Tool objectives are stated
88 (91.7)
2. Target users of tool are identified
84 (87.5)
3. User feedback about tool use/impact is prospectively collected
72 (75.0)
4. The tool was pilot-tested with target users
71 (74.0)
5. Target users were involved in tool development (i.e. interviews)
68 (70.8)
6. Sources of evidence are cited
68 (70.8)
7. Methods used to develop the tool are described
62 (64.6)
8. Evidence upon which tool is based is described (quantity/quality)
62 (64.6)
9. Tool effectiveness was assessed by full scale evaluation of impact
42 (43.8)
15
Part 2 - confirm ideal GItool features
• Two-round Delphi Internet survey of 31 guideline developers,
implementers or researchers from 10 countries
• Administered June to July 2013 with reminders at two and four weeks
• Included 9 items from cross-sectional survey and 7 items from a
meeting of 28 guideline developers who reviewed exemplar GItools
• Rate desirability of each for assessing/developing GItools on 7-point
Likert-type scale with free text option
• Consensus = 2/3 agree/strongly agree to include/exclude
16
Results: included GItool features
GItool feature
Agree or
Strongly
Agree
1. Tool objectives are stated
28 (90.3)
2. Instructions on tool use are provided
28 (90.3)
3. Target users are identified
27 (87.1)
4. Tool is based on a comprehensive search of sources for content
23 (74.2)
5. Sources of evidence are cited
23 (74.2)
6. Target users were involved in tool development (i.e. interviews)
22 (71.0)
7. The tool was pilot-tested with target users
22 (71.0)
8. Methods used to develop the tool are described
21 (67.7)
9. User feedback about tool use/impact is prospectively collected
20 (66.7)
10. Context/setting where tool developed/will be used are described
20 (66.7)
11. Evidence upon which tool is based is described (quantity/quality)
20 (64.5)
12. Methods used to evaluate the tool are described
20 (64.5)
17
Compared with cross-sectional survey
GItool feature
1. Tool objectives are stated
Agree or
Strongly
Agree
28 (90.3)
2. Instructions on tool use are provided
28 (90.3)
3. Target users are identified
27 (87.1)
4. Tool is based on a comprehensive search of sources for content
23 (74.2)
5. Sources of evidence are cited
23 (74.2)
6. Target users were involved in tool development (i.e. interviews)
22 (71.0)
7. The tool was pilot-tested with target users
22 (71.0)
8. Methods used to develop the tool are described
21 (67.7)
9. User feedback about tool use/impact is prospectively collected
20 (66.7)
10. Context/setting where tool developed/will be used are described
20 (66.7)
11. Evidence upon which tool is based is described (quantity/quality)
20 (64.5)
12. Methods used to evaluate the tool are described
20 (64.5)
18
Results: excluded GItool features
GItool feature
Conflicts of interest of developers are disclosed
Agree or Strongly Agree
Round #1
Round #2
19 (61.3)
19 (63.3)
Developers included experts in tool-relevant context/setting
---
18 (60.0)
The type of tool is specified
---
16 (53.3)
17 (54.8)
15 (50.0)
---
14 (46.7)
Developers included experts in tool content and development
19 (61.3)
13 (43.3)
Computer or mobile device application versions are available
---
13 (43.3)
15 (48.4)
12 (40.0)
---
11 (36.7)
15 (48.4)
8 (26.7)
Impact was assessed with rapid cycle testing (ie. PDSA)
---
7 (23.3)
Impact interpreted based on an implementation threshold
---
6 (20.0)
Pilot-testing was rigorous (appropriate sampling and methods)
Success factors based on tool use or evaluation are described
Full scale evaluation was rigorous (appropriate sampling, methods)
The theoretical basis or rationale for the tool is described
Impact was assessed by full scale evaluation of effectiveness
19
Part 3 - describe a sample of GItools
• GItool Framework used to describe features of GItools in a sample of
guidelines identified in AHRQ Guideline Clearinghouse in June 2012
• Eligible guidelines:
– described overall management of 8 different conditions (angina, arthritis,
asthma, breast cancer, depression, diabetes, prostate cancer)
– produced within 5 years
– by organizations having developed at least 10 guidelines
• GItools of three types were assessed for eligibility, and analysed
independently by two individuals
• Looked for information in GItools, guidelines, organization web sites
• 13 GItools identified in 149 guidelines (8.7%)
20
Results: GItool analysis
GItool feature
Type of GItool
Resource
planning (4)
Total
Implementation Evaluation
(7)
(2)
n (%)
1. Tool objectives are stated (90.3)
2
4
1
7 (53.8)
2. Instructions on tool use are provided (90.3)
4
2
2
8 (61.5)
3. Target users are identified (87.1)
4. Tool is based on a comprehensive search (74.2)
3
4
7
3
2 12 (92.3)
0 7 (53.8)
5. Sources of evidence are cited (74.2)
4
2
2
8 (61.5)
6. Target users informed tool (71.0)
4
2
1
7 (53.8)
7. The tool was pilot-tested with target users (71.0)
0
0
0
0 (0.0)
8. Development methods are described (67.7)
4
5
2 11 (84.6)
9. User feedback prospectively collected (66.7)
2
1
1
4 (30.8)
10. Context or setting is described (66.7)
4
2
1
7 (53.8)
11. Evidence is described (64.5)
2
0
1
3 (23.1)
12. Evaluation methods are described (64.5)
0
0
1
1 (7.7)
21
Summary of findings
• Engaged international guideline community
• Generated 12-item GItool Framework
• Few guidelines offered GItools
• Many GItools did not possess GItool features considered desirable
• All features ideal:
– may not be feasible given limited resource
– may not be desirable given need for access to GItools
• At minimum:
– pilot-test with target users
– share in a database that prospectively collects user feedback
22
GItool Directory (giranet.org)
23
Limitations
• How to apply the GItool Framework
– Methods for developing GItools with these features
• Examined features of few GItools
– Examine larger sample of GItools
• All GItool Framework items may not apply to different types of GItools
– Further evaluation of GItools to assess relevance
• Reflects the views of guideline experts
– Consult with guideline users
24
Ongoing research
• GItool Resource Kit
– GItool Framework: assess, adapt/adopt or endorse existing GItools
– Methods for developing new GItools: interviewed 26 developers of different
types of GItools from 9 countries who identified uniform steps and tips
• Description of GItools (#GItools/#guidelines)
– Pediatric oncology (23/29); chiropractic (7/22); dermatology (49/30);
neurology (33/26); nephrology (24/27)
– Determinants of, and trends in GItool inclusion/types/features
• Explore user preferences
– Interview and/or survey guideline users
– Users evaluate GItools with/without GItool Framework features
– Explore cognitive processes/other factors associated with GItool use
25
G-I-N Implementation Working Group
• Implementation planning checklist
– Reviewed documents describing guideline development and implementation
– Extracted information on implementation planning, strategies, instructions for
applying those strategies and GItools
– Mapped those considerations against the Guideline Development Checklist
• Trends in guideline implementation
– Systematic review of strategies used to implement guidelines (arthritis,
colorectal cancer, diabetes, heart disease) published from 2004 to 2013
– Reveal optimal strategies for guidelines with differing characteristics, and
how developers/implementers could be supported to implement guidelines
26
Acknowledgements
Co-Investigators
Melissa Brouwers, McMaster University, CA
Onil Bhattacharyya, Women’s College Hospital, CA
Collaborators / GIRAnet
Heather Buchan, Australian Commission on Safety & Quality in Health Care, AU
Andre Bussieres, McGill University, CA
Dave Davis, Association of American Medical Colleges, US
Roberto Grilli, Agency for Health and Social Care, Regione Emilia Romagna, IT
Sue Huckson, Australian & New Zealand Intensive Care Society, AU
Roberta James, Scottish Intercollegiate Guidelines Network, UK
Jorma Kormulainen, Finnish Medical Society Duodecim, FI
Ilkka Kunnamo, Finnish Medical Society Duodecim, FI
Catherine Marshall, Guideline/Health Sector Consultant, NZ
Val Moore, National Institute for Health & Clinical Excellence, UK
Philip van der Wees, Radboud University Nijmegen Medical Center, NL
Sandra Zelman Lewis, EBQ Consulting, US
27
Conclusion
• Questions for audience
– Are you interested in guidance for implementation planning?
– Are you interested in guidance for GItool assessment/development?
– In what format / how delivered?
– What additional information or resources would support implementation
planning or GItool development?
• Interested in participating?
– Evaluate GItool Resource Kit
– Evaluate Implementation Planning Checklist
– Case study of GItool development and evaluation
– Access to members for survey and/or interviews to learn about preferences
for, and use of GItools
28
Many thanks for your kind attention
[email protected]
29