Transcript Structured Assessment of Operative Reports
Structured Assessment Format for Evaluating Operative Reports (SAFE OR) in General Surgery
Ashley Vergis 1 MD, Lawrence Gillman 1 MD, Jason Park 1 Samuel Minor MD FRCSC 2 , Mark Taylor 1 MD FRCSC MD, 1 University of Manitoba, Winnipeg, Canada 2 Dalhousie University, Halifax, Nova Scotia, Canada
Medicine is an information intensive and reliant profession whose practice is
“dominated by how well information is processed or reprocessed, retrieved and communicated” Murphy et al. (1992). Good European Health Record
. Centre for Health Informatics and Multi-professional Education University College London
:
1-62.
Background
Operative dictations essential for*: Caregiver communication Quality assurance Research Medico-legal Must be comprehensive and concise!
*
Eichholz et al. (2004). Obstetrics and Gynecology
103
(2): 342-6.
Background
Multiple previous attempts to judge the quality of OR notes Third party billing criteria Comparison to consensus criteria Overall quality
appears
poor Critical details often omitted Superfluous aspects detailed
Background
However… No validated tools that examine quality of operative dictations Consequences Cannot currently measure the adequacy of an operative report Unclear if operative reports meet their stated objectives
Objective
Evaluate the ‘Structured Assessment Format for Evaluating Operative Reports’ (SAFE-OR) in General Surgery Construct validity Inter-rater agreement Internal consistency
Methods
Subjects University of Manitoba general surgery residents Novices PGY 1-3 Experienced PGY 4-6
Methods
Design View and dictate a videotaped laparoscopic sigmoid colectomy Dictation transcribed
Methods
Design Evaluated by 2 blinded, independent faculty general surgeons ‘ SAFE OR’ instrument Structured Assessment Format Adapted from CAGS clinical practice committee guidelines Global Quality Rating scale
Methods
Structured Assessment Format 19 Categorical items (yes/no) E.g., Date of Procedure 7 Ordinal items (5 point anchored Likert Scale) E.g., Findings on Exploration
Structured Assessment Form 1.
Date of Surgery 2.
Date of Dictation 3.
Patient Identifiers 4.
Name of Surgeon and Assistants 5.
Name of Anaesthetist 6.
7.
Pre-op Patient Diagnosis Post-op Patient Diagnosis 8.
9.
Procedure Performed (including side) Copies Sent to Referring Physician/Institution 10. Brief History of Present Illness (including relevant investigations) 0 No description of preoperative course or indications 1 2 Preope rative course and indications were described but some de tail was lacking 3 11. Procedure a.
Type of Anesthesia b.
Skin Preparation and Draping c.
Patient Position d.
e.
Method of Abdominal Access 0 No mention of abdominal access 1 2 Me thod of abdominal acce ss mentioned but specific details lacking Findings on Exploration of Relevant Body Cavity 0 No mention of findings on exploration 1 2 States findings of exploration but de tail/explanation lacking 3 3 YES NO YES NO YES NO YES NO YES NO YES NO YES NO YES NO YES NO 4 Comple te de scription of preoperative course and indications for procedure YES NO YES NO YES NO 4 Comple te de scription including location of incisions/trocar placements/insufflation pressure s (if applicable) 4 Comple tely explains results of exploration making note of unexpe cted findings f.
Technique of Resection, Including Tissue Mobilization 0 No description of mobilization or resection technique 1 2 Mobilization and resection techniques described but de tails regarding ope rative steps incomplete 3 4 Comple te de scription of tissue mobilization and me thod of resection including instruments used (staplers etc.) N/A
Methods
Global Quality Rating Scale 5 point anchored Likert scale 9 categories 1. Readability 2. Operative Indications 3. Operative Steps 4. Operative Findings 5. Succinctness 6. Clarity 7. Knowledge 8. Reproducibility 9. Overall Quality
Global Quality Ratings Scale
Please circle the number corresponding to the candidate’s performa nce in each category.
1. Readability of Dictation
1 Dictation was difficult to read and disorganized with incoherent flow 2 3 Dictation was relatively easy to read but at times lacked flow 4 5 Dictation was easy to read and well organized with coherent flow
2. Description of Operative Indications
1 No description of preoperative course or indications was included 2 3 Preoperative course and indication were described but some detail was lacking or inaccurate
3. Inclusion of Operative Steps
1 Dictation was incomplete as most important steps were missing 2 3 Dictation included most important steps but some detail was missing 4 4 5 Dictation includes complete description of preoperative course and indications for specific procedure performed 5 Dictation was comprehensive and included all important steps of procedure
Analysis
Construct Validity Novice and experienced groups compared (Independent samples T test) Inter-rater Agreement Intra-class correlation coefficient (ICC) >0.8 desirable* Internal Consistency Cronbach’s alpha *
Wanzel, et al. (2002). Current Problems in Surgery
39
(6): 573-659.
Results: Construct Validity
Novice (n=13) Mean Score Experienced (n=8) Structured Assessment (out of 44) 23.3
±
5.2
34.1
±
6.0
p 0.001
Global Quality (out of 45) 25.6
±
4.7
35.9
±
7.0
0.006
Results: Inter Rater Reliability & Internal Consistency
Overall ICC (95% CI) Range Structured Assessment Global Quality 0.98
(0.96 - 0.99) 0.93
(0.83 - 0.97) 0.73 - 0.90* 0.65 - 0.91
* Ordinal items, Kappa 0.73 - 1.00 for categorical items Cronbach’s Alpha 0.85
0.96
Conclusion
SAFE-OR
Satisfies construct validity Excellent inter-rater reliability Excellent internal consistency
Project Significance
SAFE-OR is the first tool that assesses the quality of an operative report in a valid and reliable manner
Project Significance
Means to more objectively measure operative report characteristics Assess degree to which reports meet stated functions (quality assurance) If unsatisfactory, may employ methods to improve reporting
Project Significance
Potential to develop methods of improving dictations in surgical education Implement SAFE-OR for assessment in training programs Formative and Summative Feedback (self and faculty) Global attributes Specific details
Project Significance
Develop, monitor and refine curriculum for operative dictation Improve communication Implications for patient care
Future Investigation
Analyze characteristics of real world operative dictations National survey (Cdn) of resident and program directors training perceptions Develop and assess specific templates for operative dictation Assess computerized dictation templates (WebSMR)* * Edhemovic et al., Annals of Surgical Oncology, 11 (10):941 –947.
Acknowledgements
Royal College of Physicians and Surgeons of Canada Medical Education Research Grant
1.
Date of Surgery 2.
Date of Dictation 3.
Patient Identifiers 4.
Name of Surgeon and Assistants 5.
Name of Anaesthetist 6.
7.
8.
9.
Pre-op Patient Diagnosis Post-op Patient Diagnosis Procedure Performed (including side) Copies Sent to Referring Physician/Institution 1 0. Brief History of Present Illness (including relevant investigations) 0 No descri ption of preoperati ve course or indications 1 2 Preope rative course and indications were described but some de tail was lacking 3 11. Procedure a.
Type of Anesthesia b.
Skin Preparation and Draping c.
Patient Position d.
e.
Method of Abdominal Access 0 No mention of abdominal access 1 2 Me thod of abdominal acce ss mentioned but specific detai ls lacki ng Findings on Exploration of Relevant Body Cavity 0 No mention of findings on expl oration 1 2 States fi ndin gs of expl orati on but de tail /explanation lacking 3 3 YES NO YES NO YES NO YES NO YES NO YES NO YES NO YES NO YES NO 4 Comple te de scripti on of preoperati ve course and indicati ons for procedure YES NO YES NO YES NO 4 Comple te de scripti on i ncl udi ng location of incisions/trocar pl acements/insuffl ation pressure s (if appl icabl e) 4 Comple tel y explai ns results of exploration making note of unexpe cted fi ndi ngs f.
Technique of Resection, Including Tissue Mobilization 0 No descri ption of mobi lization or resecti on techn ique 1 2 Mobilizati on and resecti on techn iques descri bed but de tails regarding ope rative steps i ncomplete g.
h.
Limits of Resection 0 No anatomical descri ption of structure s resected or men ti on of surgical margi ns Technique of Reconstruction 0 No mention of techn ique of reconstruction 1 1 2 Resected structure s l isted but cl ear anatomical bou ndarie s not wel l defined 2 Type of te ch nique men ti oned but steps and results of reconstruction omitted i.
Technique of Closure 0 No descri ption of closure me thod 1 2 Closure described but spe cific de tails omitted j.
k.
l.
m.
n.
o.
p.
Irrigation Drains and location Prosthesis (size and type) or other items left in location Estimated Blood Loss Dressings Applied Sponge and Instrument Count List of Specimens Sent (Micro and Pathology) 3 4 Comple te de scripti on of tissue mobil ization an d me thod of resection i ncl udi ng instruments used (stapl ers etc.) 3 4 Exact anatomical de scripti on of structure s resected i ncl udi ng all adjacent structures and margins 3 3 N/A N/A 4 Comprehen si ve description of reconstruction i ncl udi ng techni que, equipment used, and i ntegrity of repair 4 Comprehen si ve description including suture method, size and type of suture employed if appl icable YES NO N/A YES NO N/A YES NO N/A YES NO YES NO YES NO YES NO N/A N/A
Please circle the number corresponding to the candi date’s performance in each category.
1. Reada bility of Dictation
1 Dictation was difficult to read and disorganized with incoherent flow 2 3 Dictation was relatively easy to read but at times lacked flow 4 5 Dictation was easy to read and well organized with coherent flow
2. Description of Operative Indications
1 No description of preoperative course or indications was included 2 3 Preoperative course and indication were described but some detail was lacking or inaccurate 4 5 Dictation includes complete description of preoperative course and indications for specific procedure performed
3. Inclusion o f Operative Steps
1 Dictation was incomplete as most important steps were missing 2
4. Description of operative findings
1 Operative findings described were irrelevant or omitted completely 2 3 Dictation included most important steps but some detail was missing 3 Operative findings were described but some relevant detail was lacking 4 4 5 Dictation was comprehensive and included all important steps of procedure 5 Operative findings were presented in a relevant and detailed fashion
5. Succinctness of Dictation
1 Dictation was unnecessarily long with excessive inclusion of extraneous detail or truncated but lacking important elements
6. Clarity of D ictation
1 Description of included steps was vague and unintelligible 2 2 3 Dictation was relatively concise and inclusive with some extraneous detail included 3 Description of included steps was relatively clear and intelligible
7. Knowledg e
1 Clearly lacks an understanding of the procedure and relevant anatomy 2 3 Has a reasonable understanding of the procedure and relevant anatomy
8. Reproducibility of Operative Procedure
1 Recreation of operative events would be impossible from this dictation 2 3 Reader could recreate events using own knowledge to fill in gaps 1 Unacceptable dictation Major deficiencies in operative dictation skills 2 4 4 4 4
OVERALL PERFORMANCE
3 Acceptable dictation Some room for improvement 4 5 Dictation was concise but inclusive with exclusion of extraneous detail 5 Description of included steps was clear and complete 5 Demonstrates a full understanding of the procedure and relevant anatomy 5 Reader has a complete understanding of operation and could recreate procedure step by step 5 Excellent dictation Quality consistent with that expected of an attending surgeon
CAGS Clinical Practice Committ ee
Components of Operative Notes
document for gastrointestinal (GI) procedure s Date of Surg ery Date of Dictation Patient Identifi ers Name of Surg eon and Assistants Name of Ane sthetist Name of Others Present Pre and Post-op Patient Diagnos is Procedure Performed (including side) Brief Hist ory of Present Illness Including relevan t inves tigations Procedure 1.
Patient Position 2.
3.
4.
5.
6.
7.
Skin Preparation and Draping Location o f Incision (s) Findings on Explo ration of Relevant Body Cav ity Special note of un expec ted findings If canc er: extent of prim ary, lymph node and d ist ant metastases Mobili zation of Struc tures to be Operated Upon Techn ique of Resection Limits of Resection 8.
9.
Techn ique of Reconstruction Irrigation 10.
Drains and Locations 11.
Prosthesis Size and type 12.
Other Items Left and Location 13.
Techn ique of Closure 14.
Estim ated Blood Loss 15.
Dressings Applied 16.
Sponge and Instrument Count
Structured Assessment Format for Evaluating Operative Reports (SAFE OR) in General Surgery
Ashley Vergis 1 MD, Lawrence Gillman 1 MD, Jason Park 1 Samuel Minor MD FRCSC 2 , Mark Taylor 1 MD FRCSC MD, 1 University of Manitoba, Winnipeg, Canada 2 Dalhousie University, Halifax, Nova Scotia, Canada