Transcript An Evaluation of a Methodology for Specification of Clinical Guidelines at
An Evaluation of a Methodology for Specification of Clinical Guidelines at Multiple Representation Levels Student :Erez Shalom Supervisors: Prof. Yuval Shahar Dr. Meirav Taieb-Maymon
ISE Dep. Seminar 26/4/06 Talk Roadmap : Background Methods Results Conclusions Future Directions
ISE Dep. Seminar 26/4/06
Clinical Guidelines
Textual documents describing “state of the art” patient management A powerful method to standardize and improve the quality of medical care Usually specify diagnostic and/or therapeutic procedures
ISE Dep. Seminar 26/4/06 Subsections Describing patient diagnosis and treatment
ISE Dep. Seminar 26/4/06
The Need for Automation of Clinical Guidelines
Automatic support provides: Visual specification Search and retrieval Application of a GL Retrospective quality assurance However: Most GLs are text based and electronic inaccessible at the point of care
ISE Dep. Seminar 26/4/06
The Required Infrastructure
A machine-comprehensible GL representation ontology (e.g., Asbru ontology) Runtime GL application and QA tools A preliminary engine, namely , Spock was already developed in our lab by [Young,2005] Support for a gradual structuring of the GL (from text to an executable code)
ISE Dep. Seminar 26/4/06
The Structuring Process The Guideline as a text document The Guideline as a tree of plans Regimen A P Cefotetan In parallel Doxycline D P D P
Involves 2 main types of knowledge:
Procedural knowledge
– e.g. Regimen A for administer the two medications in parallel
Declarative knowledge
- e.g. 2 g IV
ISE Dep. Seminar 26/4/06
Sample GL Modeling Methods
Method
EON and SAGE, Prodigy, GLIF PRO
forma
GEM GLARE GUIDE Asbru
Knowledge Acquisition tool
A Protégé-based interface Arrezo GEM-Cutter "CG_AM" graphical interface NEWGUIDE Asbru-View, GMT, Stepper
ISE Dep. Seminar 26/4/06
The Hybrid Representation Model
Gradually structuring the GL using increasingly formal representation levels
Expert Physician Knowledge Engineer
Expert Physician Collaboration Knowledge Engineer DeGeL KB Free text Guidelines Semantic markup Semi- Structured Level Adding control structure Semi- Formal Level Formalizing to executable code //Check HGB If(HGB > 12) {...
} Formal Level //Check HGB If(HGB > 12) {...
} Implemented as part of the
Digital Electronic Guideline Library
)
DeGeL
) Used within the URUZ GL markup tool
ISE Dep. Seminar 26/4/06
Asbru- the Underlying Guideline Representation Ontology
Includes semantic Knowledge Roles (KRs) organized in KR-Classes such as:
Conditions
KR-Class (e.g., the
filter condition
, and the
abort condition
)
Plan-body
KR-Class for the GL ’s Control structures (e.g.,
sequential
,
concurrent
, and
repeating
combinations of actions or sub guidelines), GL ’s
Goals intentions
), KR-Class (e.g. process and outcome
Context
KR-Class of the activities in the GL (e.g.
actors
,
clinical-context
).
ISE Dep. Seminar 26/4/06
URUZ (I): Specification of declarative knowledge Expert physician Selects “filter condition ” knowledge role
ISE Dep. Seminar 26/4/06
URUZ (cont ’d): Specification of Procedural Knowledge Expert physician decomposing the GL into tree of plans
ISE Dep. Seminar 26/4/06
GL Specification: Core Issues
√ √ √ √ √ √ Expert Physicians (EPs) - Knowledge Engineers (KEs) collaboration Incremental Specification Treatment of Multiple Ontologies Distributed Collaboration and Sharing Text Based Source Knowledge Conversion
ISE Dep. Seminar 26/4/06
Several unresolved issues:
Definition of the necessary steps for the GL specification process Use optimally of EPs and KEs in the process Evaluation is crucial for quantify the markups quality To Achieve high quality of markups there is a Need for: • An overall process of guideline specification • A complete evaluation methodology
ISE Dep. Seminar 26/4/06
Talk Roadmap
√ Background Methods Results Conclusions Future Directions
ISE Dep. Seminar 26/4/06 The Overall Process of Guideline Specification The activities in the markup process include three main phases : 1) Preparations
before
the markup activities 2) Actions d
uring
the Markup activities 3)
After
Markup activity
ISE Dep. Seminar 26/4/06
A Methodology Specification of Clinical Guidelines
Before Markup During Markup After Markup
1.
KE Choose Specification Language EP 2.
KEs Instructing the EPs KE KE EP 3.
Choosing together GL for Specification KE EP 4.
together an Ontology Specific Consensus KE EP 5.
EPs Training in the Markup Tool EP 6.
together Gold Standard Markup EP 7.
EPs Classify the GL EP 8.
EPs Perform Markup KE EP 9.
Evaluating together the Markups
Creating a consensus is a crucial, mandatory step before markup
ISE Dep. Seminar 26/4/06
The Importance of Using an Ontology-Specific Consensus (OSC)
An OSC is a structural document that describes schematically the clinical directives of the GL Described by the semantic of the specification ontology Prevent disagreement and a great deal of variability among the EPs
ISE Dep. Seminar 26/4/06
Methodology for Creation of OSC
The OSC is created in a iterative fashion by performing the following steps 1.
First, we create a preliminary structure of the clinical pathway 2.
The KE adds procedural, control structure 3.
The KE adds declarative concepts for each defined step 4.
After some iteration of steps 2 and 3, an OSC is formed
ISE Dep. Seminar 26/4/06
The second stage in forming a consensus
Patient Treatment and evaluation
NO
Outpatient Treatment and evaluation (parallel) Outpatient Treatment Outpatient Evaluation and follow up
See 1.8
is PID severe? (*)
See 1.7
YES
Hospitalization and discharged (sequential) Hospitalization
See 1.2
Post Discharged
See 1.5
ISE Dep. Seminar 26/4/06
Evaluation Design
Considered some specific Criteria :
Amount of Expertise
The acquired knowledge domain
The Ontology Specific Consensuses
The Gold Standard markup for each GL
The Markups for each GL
The Evaluation of markups
ISE Dep. Seminar 26/4/06
Evaluation Design (cont ’d)
Three GLs in different domains were used :
Pelvic Inflammatory Disease (PID) Chronic Obstructive Pulmonary Disease (COPD) Hypothyroidism(HypoThyrd)
ISE Dep. Seminar 26/4/06
Evaluation Design (cont ’d)
EP Hypo Thyrd COPD PID KE EP Hypo Thyrd COPD PID
GL Sources One Ontology Specific Consensus For each GL
EP EP Hypo Thyrd COPD PID COPD PID
Markup
Hypo Thyrd
Markup Markup Markup Markup Markup
Two markups for each GL Each markup is evaluated compare to the Gold Standard
KE EP Hypo Thyrd COPD PID
Gold Standard Gold Standard Gold Standard
One Gold Standard Markup for each GL
EP
ISE Dep. Seminar 26/4/06
Research Questions
Is markup feasible by EPs? Is there a difference between the EPs editing the same GLs , and same EPs editing difference GLs?
Is there a difference between the KRs across all EPs?
Is there a difference in the amount of errors when using different OSC?
ISE Dep. Seminar 26/4/06
Evaluation of markups
Subjective Measures
Questionnaires were administered for finding the EPs attitude regarding the specification process
Objective Measures –
in two scales (compared to the GS): * Completeness of the markup * Correctness of the markup
ISE Dep. Seminar 26/4/06
The Objective Measures - Completeness
GS Markup A B C
ISE Dep. Seminar 26/4/06
The Objective Measures - Correctness
* Clinical Measure (CM)
– measure the clinical correctness
*
of the content
Asbru Semantic Measure (ASM
) - measure the semantics correctness of the content ( Asbru semantic in our case)
ISE Dep. Seminar 26/4/06
Resolution of Measure
Mean (weighted) Quality Score (MQS) for: GLs - to find common trends in a GL, and in all GLs EPs - to find trends in between the markups of the EPs across the same GL and between GLs KRs - to find trends in a specific KR type and common trends across KRs and KR classes across one markup, GL and in all GLs
ISE Dep. Seminar 26/4/06
The Objective Measures – Types of Errors
General errors classified
into two types, and thus into two corresponding scales: Clinical errors: Clinical content not accurate Clinical semantics not well specified Clinical content not complete.
Asbru semantics errors: Asbru semantics content not accurate Asbru semantics content not well specified The content does not includes mappings to standard terms The necessary knowledge is not defined in the guideline knowledge when it should be.
ISE Dep. Seminar 26/4/06
The Objective Measures – Types of Errors
Specific errors
for each KR Type a specific error, for example : Conditions /Intentions KRs: There are no And/Or operators between the different criteria.
Simple Action Plan-Body Type: Has no text content describing the plan Has no single atomic action semantics with clear specification and description for the action to be performed. Plan Activation Plan-Body Type: Plan name is not defined Defined plan does not exist in DeGeL.
ISE Dep. Seminar 26/4/06
The Markup-Evaluation Tool
ISE Dep. Seminar 26/4/06
The Markup-Evaluation Tool (Cont ’d)
ISE Dep. Seminar 26/4/06
The Markup-Evaluation Tool (Cont ’d)
ISE Dep. Seminar 26/4/06
Talk Roadmap
√ Background √ Methods Results Conclusions Future Directions
2 1 ISE Dep. Seminar 26/4/06
Results – Subjective Measures
Purpose Stage Result Finding the Aspects that most helped the EPs' when creating an OSC after creating the OSC Using their medial knowledge and their understanding of the specification ontology (Asbru, in our case) Vs. specification Tools Finding the Aspects that most helped the EPs' making a markup after markup Specification Tools is considered as more helpful
3
Finding how well The EPs Understand Asbru KRs before markup Declarative KRs are more easy to understand (such as filter condition)
4
Finding what were the difficulties of the EPs' in structuring the Asbru KRs after markup Procedural KRs are more easy to structure (such periodic plan)
5
System Usability Scale (SUS) for URUZ after markup SUS=47 ; Not Usable!
• Non significant correlation between results 1 and 2 • Significant correlation between results 3 and 4
ISE Dep. Seminar 26/4/06
Results – Objective measures
Number of specified plans: Measures Summary: • Mean Completeness for all markups of EPs of 91% •All markups of EPs has significant (P<0.05) proportion of scores of 1 higher than 0.33 (some even higher then 0.75) Markup is feasible by EPs
1 2 3 4
ISE Dep. Seminar 26/4/06
Results –Difference between EPs
Issue significant (P<0.05) nonsignificant (P>0.05) Difference between the proportions of completeness measure between EPs editing the same GLs Difference in correctness measure between EPs editing the same GLs in most GLs (except the Hypo) Difference in Correlation measure between EPs editing the same GLs in most GLs (except the PID) Difference in correctness measure between different GLs editing the same EPs
√ √ √ √ •Any EP can perform markup with high completeness •There is wide variability between the EPs in the correctness measure with a range of [0.13,0.58] on a scale of [-1,1]
ISE Dep. Seminar 26/4/06
Results – Difference between KRs
There was significant difference (P<0.05) between homogenous groups of KRs EPs has difficulty to structure procedural KRs than declarative ones
ISE Dep. Seminar 26/4/06
Results – Types of errors
0.6
0.5
0.4
0.3
0.2
0.1
0 199 360
0.55
112 360
0.31
87 360
0.24
Markup Error Rate(MER) per KR, per EP
79 194
0.41
61 194
0.31
18 194
0.09
12 9 98 98
0.12
0.09
3 98
0.03
PID Total Ontological Clinical COPD
GLs
HypoThyrd
No
.
of No
.
of errors KRs in GL i in GL i
* 2 = MER
i
. The differences in
total
between the three GLs were highly significant in a proportion test (P<0.001) The more detailed and structured the OSC was, the lower the total number of errors committed by the EPs for each KR
ISE Dep. Seminar 26/4/06
Talk Roadmap
√ Background √ Methods √ Results Conclusions Future Directions
ISE Dep. Seminar 26/4/06
Four main aspects :
Creation of an Ontology-Specific Consensus (OSC) The essential aspects needed to learn to support the specification process by EPs The medical and computational qualifications needed for specification The characteristics of the KA tool needed for this kind of specification
ISE Dep. Seminar 26/4/06
Creation of an Ontology-Specific Consensus (OSC)
Should be made as detailed as possible, including all relevant procedural and declarative concepts The OSC is independent of the specification tool Saving the OSCs in an appropriate digital library for re-using and sharing
ISE Dep. Seminar 26/4/06
The Essential Aspects Needed to Learn to Support the Specification Process by EPs
creating an OSC and performing the markups are two different tasks which require teaching two different aspects Teaching the “difficult” KRs in particular, the procedural KRs Short test should be administered before the EPs perform markups A help manual and a small simulation of marking up a GL should be included in the teaching session
ISE Dep. Seminar 26/4/06
The Medical and Computational Qualifications Needed for Specification
Senior EPs and KEs together should work on the tasks of selecting a GL for specification and making the OSC Any EP (senior, non-senior or a general physician) can structure the GL's knowledge in a semiformal representation completely To specify it correctly, a more available EP should be selected, perhaps from among residents, interns or even students
ISE Dep. Seminar 26/4/06
The Characteristics of the KA Tool Needed for This Kind of Specification
A robust, graphical, highly usable framework is needed More intuitive, graphic, user friendly interfaces should be used for acquiring the “difficult” KRs , especially the procedural ones Need to bridge the gap between the initial structuring of the EP and the full semantics of the specification language
GESHER A Graphical Framework for Specification of Clinical Guidelines at Multiple Representation Levels
ISE Dep. Seminar 26/4/06
Limitations and Advantages of the research
Small of the number of EPs and GLs, But, in fact, 196 sub-plans and 326 KRs in total were structured by all of the EPs together in all markups Lack of careful measurement of the required time , but, obtain more realistic results, since the interaction with most of the EPs took place in their own "playground"
ISE Dep. Seminar 26/4/06
Talk Roadmap
√ Background √ Methods √ Results √ Conclusions Future Directions
ISE Dep. Seminar 26/4/06
The GESHER ’s Main Features
User friendly graphical client application Support specification at multiple representation levels Support to multiple specification languages (GL ontologies) Access centralized resources such as DeGeL and a knowledge base
ISE Dep. Seminar 26/4/06
GESHER: Semi-Structured Level
The Hybrid Ontology Tree showing KRs at all representation levels
ISE Dep. Seminar 26/4/06
GESHER(II) :Semi-Formal Widgets
ISE Dep. Seminar 26/4/06
Talk Roadmap
√ Background √ Methods √ Results √ Conclusions √ Future Directions
ISE Dep. Seminar 26/4/06
Summary
The need for gradual GL specification Making an ontology-specific consensus as first step Use a well defined methodology for the overall process Markup is feasible by EPs Any EP can perform markup with high completeness We should use methodology for increase quality of markups Use GESHER as the new framework for specification Ongoing new research is being conducted (Pre-Eclampsia GL) based on this research results
ISE Dep. Seminar 26/4/06
Research Publications
2005 תויאופר מ " מל ילארשיה סנכה , גוציי תומר ' סמב תיפארג הדובע תביבס Shalom, E. and Shahar Y. (2005). A Graphical Framework for Specification of Clinical Guidelines at Multiple Representation Levels. AMIA Annual Fall Symposium, Washington DC, USA 2006 תויאופר מ " מל ילארשיה סנכה ?
יאופר עדי תונבהל םילגוסמ םיאפור םאה Shalom E, Shahar Y, Young O, Bar G, Taieb-Maimon M, Yarkoni A, B.Martins S, Vaszar L, K.Goldstein M, Liel Y, Leibowitz A, Marom T, and Lunenfeld E. (2006) A Methodology for Evaluation of A Markup-Based Specification of Clinical Guidelines Submitted to AMIA , Washington DC, USA Shalom E, Shahar Y, Young O, Bar G, Taieb-Maimon M, Yarkoni A, B.Martins S, Vaszar L, K.Goldstein M, Liel Y, Leibowitz A, Marom T, and Lunenfeld E.(2006) The Importance of Creating an Ontology-Specific Consensus Before a Markup-Based Specification of Clinical Guidelines , Submitted to ECAI06 ,Tronto, Italy JAMIA journal paper is in preparation
ISE Dep. Seminar 26/4/06
Acknowledgments
Prof. Yuval Shahar and Dr. Meirav Taib-Maymon All Medical informatics research center members Our colleagues at Soroka ’s university medical center : Prof. Eitan Lunenfeld, Dr. Avi Yarkoni, Dr. Guy Bar, Prof. Yair Liel and Dr. Tal Marom Our colleagues at Stanford and VA hospital: Drs. Mary Goldstein, Susana Martins, Lawrence Basso, Herbert Kaizer, Laszlo Tudor Vaszar NLM award No LM-06806 Contact info : [email protected]
Visit our web site : http://medinfo.ise.bgu.ac.il/medlab/
ISE Dep. Seminar 26/4/06
Questions?
ISE Dep. Seminar 26/4/06
Methodology for creating an Ontology-Specific Consensus
EPs create a Clinical consensus :
Give Doxycycline 100 mg orally or IV every 12 hours Metronidazole 500 mg IV every 8 hours
Plus EPs and KE adds procedural knowledge :
Order :parallel
Ontology- specific consensus
Doxycycline Metronidazole
EP+KE add
declarative
knowledge
: Filter condition for drug : Is patient not sensitive to Doxycycline and the drug available?
ISE Dep. Seminar 26/4/06
Textual Source
Parenteral Regimen A
Cefotetan
2 g IV every 12 hours
OR Cefoxitin
2 g IV every 6 hours
PLUS
Doxycycline
100 mg orally or IV every 12 hours.
Parenteral Regimen B
Clindamycin
900 mg IV every 8 hours
PLUS
Gentamicin
loading dose IV or IM (2 mg/kg of body weight) followed by a maintenance dose (1.5
substituted. mg/kg) every 8 hours. Single daily dosing may be
Alternative Parenteral Regimens
Ofloxacin
400 mg IV every 12 hours
OR Levofloxacin
500 mg IV once daily
WITH or WITHOUT
Metronidazole
500 mg IV every 8 hours
OR Ampicillin/Sulbactam
3 g IV every 6 hours
PLUS
Doxycycline
100 mg orally or IV every 12 hours.
ISE Dep. Seminar 26/4/06
The first stage in forming a consensus
Inpatient treatment -
Give one of the following regimens
Parenteral Regimen A
Cefotetan
2 g IV every 12 hours
OR Cefoxitin
2 g IV every 6 hours
PLUS
Doxycycline
100 mg orally or IV every 12 hours.
Parenteral Regimen B
Clindamycin
900 mg IV every 8 hours
PLUS
Gentamicin
loading dose IV or IM (2 mg/kg of body weight) followed by a maintenance dose (1.5
mg/kg) every 8 hours.
Single daily dosing may be substituted.
Alternative Parenteral Regimens
Ofloxacin
400 mg IV every 12 hours
OR Levofloxacin
500 mg IV once daily
WITH or WITHOUT
Metronidazole
500 mg IV every 8 hours
OR Ampicillin/Sulbactam
3 g IV every 6 hours
PLUS
Doxycycline
100 mg orally or IV every 12 hours.
ISE Dep. Seminar 26/4/06
The second stage in forming a consensus
Parenteral Regimen A (parallel)
Cefotetan or Cefoxitin Cefotetan
2 g IV every 12 hours
OR Cefoxitin
2 g IV every 6 hours
PLUS
Doxycycline
100 mg orally or IV every 12 hours.
OR
Inpatient treatment (sequential)
note :give any
one
of the regimens OR OR
Parenteral Regimen B (parallel)
Clindamycin
900 mg IV every 8 hours
PLUS
Gentamicin
loading dose IV or IM (2 (1.5
mg/kg of body weight) followed by a maintenance dose mg/kg) every 8 hours. Single daily dosing may be substituted.
Alternative Parenteral Regimens (parallel)
(Ofloxacin or Levofloxacin) +/ (Metronidazole or Ampicillin/ Sulbactam) Ofloxacin
400 mg IV every 12 hours
OR Levofloxacin
500 mg IV once daily
WITH or WITHOUT
Metronidazole
500 mg IV every 8 hours
OR Ampicillin/Sulbactam
3 g IV every 6 hours
PLUS
Doxycycline
100 mg orally or IV every 12 hours.
ISE Dep. Seminar 26/4/06
The third stage in forming a consensus
ISE Dep. Seminar 26/4/06
Results – Subjective measures
ISE Dep. Seminar 26/4/06
ISE Dep. Seminar 26/4/06
ISE Dep. Seminar 26/4/06
Results – Objective measures
• Mean Completeness for all markups of EPs of 91% •All markups of EPs has significant (P<0.05) proportion of scores of 1 higher than 0.33 (some even higher then 0.75
Markup is feasible by EPs
ISE Dep. Seminar 26/4/06