Transcript http://www.hrbcentreprimarycare.ie/ppt/FaheyBirmingham.ppt

Royal College of Surgeons in Ireland
Coláiste Ríoga na Máinleá in Éirinn
Implementing Clinical Prediction Rules in primary
care: evidence, challenges and possible solutions
Tom Fahey
Division of Population Health Sciences
Outline of talk
•
•
•
•
Background
Evidence concerning implementation of CPRs
Barriers to implementation
Solutions to implementation
– Cochrane Register of CPRs in primary care
– Computer based clinical decision support systems
(CDSSs)
Division of Population Health Sciences
(1) Background
• Definitions
• Stages of development
• Implementation
Division of Population Health Sciences
Definitions
• Clinical Prediction Rule
– Clinical tools that quantify the contribution of
• Patient History
• Physical Examination
• Diagnostic Tests
– Stratify patients diagnosis
• Probability of having target disorder.
– Outcome can be in terms of diagnosis, prognosis,
referral or treatment
Division of Population Health Sciences
Example of a CPR: The Centor Score
(2) Evidence concerning
implementation of CPRs
Division of Population Health Sciences
Stages of development of CPR
Figure 2. Development of a Clinical Decision Rule
Step 1. Derivation
Identification of factors
with predictive power
Step 2. Validation
Step 3. Impact Analysis
Evidence of reproducible accuracy
Narrow Validation
Application of a rule in a
Similar clinical setting and
Population as Step 1
Broad Validation
Application of rule in
multiple clinical settings
and varying prevalence
and outcomes of disease
Evidence that rule change
physician behaviour and
improves patient outcomes
and/or reduces costs
Level of Evidence
4
3
2
1
Division of Population Health Sciences
Impact analysis studies
• Some clinical examples
• Evidence base
• Issues that need to be considered
Division of Population Health Sciences
CPR pigmented skin lesions
Division of Population Health Sciences
Canadian C-spine Rule
Division of Population Health Sciences
CPR for deep venous thrombosis
(DVT)
Division of Population Health Sciences
Chest pain triage CPR
Division of Population Health Sciences
Ottawa ankle rule
Division of Population Health Sciences
Primary prevention of
Cardiovascular disease
Division of Population Health Sciences
Implementation studies- stage 3
(level 1 evidence)
Domain
Examples
Impact
Trauma
Canadian C-Spine rule
Ottawa ankle rule
Diminished imaging
Missed fracture
Other adverse events
Cardiovascular disease
Chest pain
Safety of triage decisions
DVT CPR including NPT
Referral to secondary care
Incidence DVT in low risk
patients
Process of care- CVS drugs
prescribed
Prevention of CVS event
Primary prevention risk
estimation
Infectious disease
Centor score
UTI risk score
Dermatology
Melanoma score
Antibiotic use (reduced)
Antibiotic use (lower than
empirical but not delayed)
Division of Population Health Sciences
Patient’s symptoms
Ratio of benign/malignant
detected
Issues that arise from
Implementation studies
Issue
Consequence
Target condition
“High stakes” condition
Resource use
Clustering in specific
clinical domains
Methodology
Cluster RCT
“Before/After” study
Complex
Time-consuming
Costly
Rare
Comparison strategy
Usual care
Guidelines
Near patient test
Combinations
Context/Country specific
Limits generalisability
May get outdated
Outcome
Process of care
Patient centred outcome
Context specific
Costly and time
consuming
Division of Population Health Sciences
(3) Barriers to implementation
• Cognitive process in relation to
diagnosis/prognosis
• Challenges
Division of Population Health Sciences
Diagnostic stages & strategies
.
Division of Population Health Sciences
Focussed clinical assessment
Division of Population Health Sciences
Strategies used for refining
diagnosis
Division of Population Health Sciences
CPRs in diagnostic process
• One of several “refinement” techniques
• Not always appropriate to use in terms of clinical
context
• Not always available
Division of Population Health Sciences
Barriers to implementation
• Cognitive process in relation to diagnosis/prognosis
• Challenges
Division of Population Health Sciences
Challenges to Implementation
Sources of
Behaviour
Change
Barriers to
Innovation
Knowledge
Attitudes
Behaviour
External Barriers
Lack of familiarity
Lack of agreement
1. Time Investment
1. Uncertain
application
1. Lack of time
2. Unclear
applicability
3. GP Software
companies
2. Inadequate
infrastructure
Adapted from Lang et al. 2007
2. Resources
4. Poor integration
into workflow
Division of Population Health Sciences
(4) Solutions to implementation
• Cochrane register of CPRs in primary care
• Implementation of CPRs with computer-based
clinical decision support systems
Division of Population Health Sciences
Division of Population Health Sciences
Division of Population Health Sciences
Work package #2
• Develop a web-based register of CPRs for use in
primary care
Problems with developing and
maintaining the register
• Problems associated with developing and
maintaining the register
– Interchangeable terminology for CPRs
– No Medical Subject Heading (MeSH) term for CPRs
– Over two million articles are published every year
Division of Population Health Sciences
Problems with developing and
maintaining the register
Haynes and the Hedges Team have developed search tools to
identify CPRs.
30 journals included on the
register
Academic Emergency Medicine
Family Medicine
American Family Physician
Family Practice
American Journal of Medicine
Journal of American Medical Association
Annals of Emergency Medicine
Journal of the American Board of Family Medicine
Annals of Family Medicine
Journal of Clinical Epidemiology
Annals of Internal Medicine
Journal of Family Practice
Annals of Medicine
Journal of Internal Medicine
Annual Review of Medicine
Lancet
Archives of Internal Medicine
Medical Care
BMC Family Practice
Medical Decision Making
British Medical Journal
Medicine
British Journal of General Practice
New England Journal of Medicine
Canadian Family Physician
Public Library of Science Medicine
Canadian Medical Association Journal
Primary Care
Cochrane Database Systematic Reviews
Scandinavian Journal of Primary Health Care
Search filter for CPRs in primary
care
• Manually searched 30 journals relevant to primary care for the
year 2008 (‘reference standard’)
• 7 individual electronic searches of the 30 journals (each filter
treated as ‘diagnostic tests’)
• Test accuracy analysis: Sensitivity and specificity
• Aim: to maximise sensitivity
Division of Population Health Sciences
Database
PubMed
Filter name
Haynes Broad Filter
(HBF)
Haynes Narrow Filter
(HNF)
McGrath/Murphy
Broad Filter (MMBF)
Filter search string
(predict*[tiab] OR predictive value of tests[mh] OR scor*[tiab] OR observ*[tiab] OR
observer variation[mh])
(validation[tiab] OR validate[tiab])
EBSCO host
McGrath/Murphy
Narrow Filter
(MMNF)
((predict* N3 rule* OR predict* N3 model OR predict* N3 models) OR (decision* N3
rule*))
PubMed
Teljeur/Murphy
Inclusion Filter 26
item (TMIF-26)
PubMed
Teljeur/Murphy
Inclusion Filter 22
item (TMIF-22)
"clinical prediction" OR "clinical model*" OR "clinical score*" OR "decision rule*" OR
"diagnostic accuracy" OR "diagnostic rule*" OR "diagnostic score*" OR "diagnostic
value" OR "predictive outcome*" OR "predictive rule*" OR "predictive score*" OR
"predictive value" OR "predictive risk*" OR "prediction outcome*" OR "prediction
rule*" OR "prediction score*" OR "prediction value*" OR "prediction risk*" OR "risk
assessment" OR "risk score*" OR "validation decision*" OR "validation rule*" OR
"validation score*" OR (derivation AND validation) OR (sensitivity AND specificity) OR
(symptoms AND signs)
(clinical[tiab] AND predict*[tiab]) OR (clinical[tiab] AND model*[tiab] ) OR
(clinical[tiab] AND score*[tiab]) OR (decision [tiab] AND rule*[tiab]) OR (derive*[tiab]
AND validat*[tiab]) OR (diagnos*[tiab] AND accura*[tiab]) OR (diagnos*[tiab] AND
rule*[tiab]) OR (diagnos*[tiab] AND score*[tiab]) OR (diagnos*[tiab] AND value[tiab])
OR (predict*[tiab] AND outcome*[tiab]) OR (predict*[tiab] AND rule*[tiab] OR
(predict*[tiab] AND score*[tiab] ) OR (predict*[tiab] AND validat*[tiab]) OR
(predict*[tiab] AND value*[tiab]) OR (risk*[tiab] AND assessment*[tiab]) OR (risk[tiab]
AND score*[tiab]) OR (sensitivity[tiab] AND specificity[tiab]) OR (symptoms[tiab]
AND signs[tiab]) OR (validat*[tiab] AND decision*[tiab]) OR (validat*[tiab] AND
rule*[tiab]) OR (validat*[tiab] AND score*[tiab]) OR (predict*[tiab] AND risk*[tiab])
PubMed
Teljeur/Murphy
Exclusion Filter
(TMEF)
PubMed
EBSCO host
((predict* N3 rule* OR predict* N3 model OR predict* N3 models) OR (decision* N3
rule*) OR (TX validat*))
(allele OR amino OR animal OR apoptosis OR chromosome OR congenital OR dental
OR dna OR endogenous OR endothelial OR epithelial OR mammalian OR mice OR
molecule OR molecular OR mouse OR mutate OR mutation OR necrosis OR
pathogenesis OR phosphorylation OR polymorphism OR receptor OR signal OR species
Results
Manual ‘reference standard’ search retrieved 6344 articles, 41 of which were
CPRs
Filter name
N
articles
retrieved
N CPRs
retrieved
Sensitivity (%)
Specificity (%)
1251
89
31
12
76
29
81
99
264
23
56
96
63
16
39
99
Teljeur Murphy Inclusion Filter-26 item
2432
39
95
62
Teljeur/Murphy Inclusion Filter-22 item
693
34
83
90
3589
24
59
43
Haynes Broad Filter
Haynes Narrow Filter
McGrath/Murphy Broad Filter
McGrath/Murphy Narrow Filter
Teljeur/Murphy Exclusion Filter
Number of articles in PubMEd
MEDLINE versus the final search filter applied
to 30 primary care journals (1966 – 2008)
Year of publication
Creating the International
Register
2008-1980 resulted in 252 CPRs relevant to primary care
Years
Average articles retrieved
per year
Average CPRs for
primary care identified
per year*
2008 – 2000
2622
25
1999-1990
2328
3
1989-1980
2243
2
* Still awaiting inter-library loans
Clinical Domains – ICPC2
Percentage articles retrieved for each clinical domain
0
General and Unspecified
Hematologic System
Digestive
Eye
Ear
Cardiovascular
Musculoskeletal
Neurological
Psychological
Respiratory
Skin
Endocrine/Metabolic/Nutritional
Urological
Pregnancy/Childbearing/Family…
Female Genital
Male Genital
%
10
%
20
%
30
%
40
%
50
%
Clinical Domains – ICPC2
Respiratory System
Respiratory System
N=41
Strep throat
12
Pneumonia
8
Asthma
6
Respiratory symptom/complaint other
3
Chronic obstructive pulmonary disease
2
Cough
2
Respiratory infection other
2
Breathing problems , other
1
Influenza
1
Respiratory disease other
1
Tonsillitis acute
1
Upper respiratory infection acute
1
Sinusitis acute/chronic
1
Quality assessment
CPRs (derivation), 2008, n=18
All outcome events clearly defined?
All predictors clearly defined?
All important predictors present in a significant proportion
of the study population?
Were all important predictors included in the derivation
process?
Yes
No
Unreported
Clinically sensible?
Adequate sample size? (including outcome events)
Were those assessing the presence of predictors blinded to
the outcome event?
Were those assessing the outcome event blinded to
presence of predictors?
0
2
4
6
8
10
12
14
16
18
Division of Population Health Sciences
Quality assessment
CPRs (validation), 2008, n=20
Do patients represent a wide spectrum of severity of
disease?
Were patients selected in an unbiased fashion?
Yes
No
Was there ≥80% follow up of those enrolled?
Unreported
Were those assessing the presence of predictors blinded to
the outcome event?
Were those assessing the outcome event blinded to
presence of predictors?
0
2
4
6
8
10
12
14
16
18
20
Division of Population Health Sciences
Electronic CPR Register
• International electronic register of CPRs
• Searchable electronic resource
• Web based user friendly access
• Manage and maintain register
Division of Population Health Sciences
Registry
1
2
3
4
5
6
Division of Population Health Sciences
Ongoing Work
Search
other
databases
Register
Expert
contact
Secondary
sources
•
•
•
•
Embase
Cochrane
Medion
LILACS
• Author’s databases
• Published
• Unpublished
• Evidence Based
Medicine (BMJ )
• Rational Clinical
Examination (JAMA)
Register of CPRs- current status
• Identified the search filter with the highest level of
sensitivity to search the MEDLINE database using the
PubMed interface
• Significant reduction in the items to be searched
• Approximately 252 CPRs across several clinical domains
(2008-1980)
• Still awaiting inter-library loans
• Further resources are being searched as part of ongoing
work
Division of Population Health Sciences
Solutions to implementation
• Cochrane register of CPRs in primary care
• Implementation of CPRs with computer-based
clinical decision support systems (CDSSs)
Division of Population Health Sciences
Implementation of evidence
Division of Population Health Sciences
Division of Population Health Sciences
Division of Population Health Sciences
Division of Population Health Sciences
Division of Population Health Sciences
Clinical decision support system
• Clinical decision support system (CDSS)
– Systems that are designed to improve clinical
decision making
• Key points
–
–
–
–
Integrated with the electronic patient record
Available at the point of care
Computerised knowledge base
Provide patient-specific content
Division of Population Health Sciences
Implementation
Clinical Domain : Pharyngitis
CPR : Centor Score
• CDSS based on Bayesian reasoning
– Reasoning engine 
• Software Algorithm
• Combining Clinical Prediction Rules in registry to patient data
– Communication mechanism 
• Input : Electronic Patient Record
• Output : Diagnostic and therapeutic recommendations
Division of Population Health Sciences
Example of a CPR: The Centor Score
Division of Population Health Sciences
CDSS: CENTOR Score
Division of Population Health Sciences
CDSS : Risk Stratification
Division of Population Health Sciences
CDSS Management Recommendations
Division of Population Health Sciences
CDSS Prescribing Recommendations
Patient Specific
Division of Population Health Sciences
The TRANSFoRm Project
• “Translational Research and Patient Safety in Europe”
• 5 Year EU funded research project – just started
• 17 Participant bodies including the HRB Centre
• Leading Work Package 4 – “Decision Rules and Evidence”
• Bridge clinical research and primary care practice – will
involve validation in Irish GP sites
Division of Population Health Sciences
The TRANSFoRm Project
Division of Population Health Sciences
TRANSFoRm Work Packages
Division of Population Health Sciences
TRANSFoRm
Services
1 CPR Repository
Clinical Prediction
Rules Service
2 Distributed
GP EHRs
With CDSS
3 Research Study Designer
Study Criteria
Design
CP Rules
Manager
CP Classifier
5 CPR Data Mining
and Analysis
CPR Analysis &
Extraction Tool
Find Eligible Patient
(
4 Research Study Management
Recruit Eligible Patient
Study Data Management
Division of Population Health Sciences
TRANSFoRm
• “Improvements in safety performance have the potential to
benefit both patients and providers and to align their
interests” - RAND Report – Greenberg, 2010
• ICT systems based on:
– electronic CPR repositories
– integrated with EHR to provide CDSS
– integrated with clinical research – “self learning” CPRs
– reduce possibility of diagnostic error
– improve patient safety
Division of Population Health Sciences
(4) Implementation of CPRstwo complementary approaches
Register of CPRs Cochrane Primary Health Care Field
• Implement International Register for CPRs
– Searchable across clinical domains
– Public web based user friendly access
– Maintained and updated
Computer based clinical decision support systems (CDSSs)
• Implement CDSSs
– Knowledge base
– Software algorithms [Bayesian reasoning]
– Integrated in electronic patient record
– Patient-specific recommendations
Division of Population Health Sciences
Conclusions
• Evidence concerning implementation of CPRsneeds strengthening
• Barriers to implementation- identifiable and
remediable
• Solutions to implementation
– Cochrane Register of CPRs in primary care
– Computer based clinical decision support systems
(CDSSs)
Division of Population Health Sciences
Acknowledgements
• Colleagues in HRB Centre for Primary Care
Research
• Health Research Board (grant HRC/2007/1)
Division of Population Health Sciences
Division of Population Health Sciences
Additional slide
Implementation
Division of Population Health Sciences
Strategies to overcome barriers
to implementation
Division of Population Health Sciences
©2006 by American College of Physicians