Transcript ProMIS by Haptica - Trinity College, Dublin

Efficient rigid body
motion tracking with
applications to
human psychomotor
performance
assessment
“Surgeons who trained on simulations
performed the operation 29% faster and
made 7 times fewer errors”.
Annals of Surgery 2002
Gerard Lacey
Computer Vision and Robotics Group
Dept. of Computer Science
Trinity College Dublin
www.haptica.com | tel. 617 342 7270 | email. [email protected]
Overview of Talk
• Context for research – Haptica Ltd
• Laparoscopic Surgery Intro
• Solutions to Surgical Training
• Computer vision tracking
• Live multimedia annotation of training video
• The results
• The current position
• The Future
Haptica ProMIS May2006
Page 2
Inspiration for ProMIS
• Previous Research
• Mobile Robotics for Frail Visually
Impaired Elderly
• Bayesian Belief Networks for multi-sensor
classification / context sensitive reasoning
• Haptica Ltd formed in 2000
• Focus on human performance
enhancement in critical tasks
•
•
•
•
•
Haptica ProMIS May2006
Page 3
Guidance for mobility
Computer Assisted Docking
Fraud detection in credit card payments
Semantic Web search of patent databases
Laparoscopic Surgery Training
Laparoscopic Surgery
•
Laparoscopy – Minimally Invasive Surgery
•
•
•
•
Key Challenges
•
•
•
•
•
Page 4
Loss of dexterity
Fulcrum effect
Loss of sense of touch
Slower than conventional surgery
Environment for learning
•
•
•
•
Haptica ProMIS May2006
Surgeons operate with long narrow instruments
Watch their movements on TV monitors
Rapidly growing due to patient demand and
cost savings in reduced patient stays
Apprentice model
See one, do one, teach one
Patients used for learning
No objective assessment
Current Training Methods
• Current Training Tools
• Patients
• Physical simulators - “Box Trainers”
• Virtual Reality and Full Haptic Simulators
Physical Sim
System behaves like abdomen
Realistic tactile feedback
Realistic visual feedback
Use the real tools
Team approach to procedure
Simulate specific surgical procedure
Assess Trainee’s performance
Use with computer based training
Playback a video of their operation
Simulate affect on of errors
Simulate unusual pathologies
Haptica ProMIS May2006
Page 5
VR Sim
Objective Skills Measurement in Medicine
• Human observation of bench tests
• Resnick – U Toronto (OSATS)
• Track hand movements to measure surgical skills
• Darzi – Imperial College (ICSAD)
• Linking skills to patient outcomes
• VR to OR trials
• Key understanding of the skills to be learned
• Cognitive X% + Affective Y% + Psychomotor Skill Z%
• Surgery mainly 75% X & 25% Z with Z higher in minimally
invasive surgery
Haptica ProMIS May2006
Page 6
Key Skills in surgery
• Key indicators of skill acquisition
• The objective is Proficiency
• The time proxy – How fast can it be done?
• The repetitions proxy – How many times has it been done?
• The errors proxy – have any of these listed errors occurred
while the task was being completed?
• Surgical Finesse – smooth movement /
economy of movement
• Path of surgical instruments in space
Novice
Haptica ProMIS May2006
Page 7
Expert
Measuring Economy of Movement
Stereo Cameras Track Instruments
Haptica ProMIS May2006
Page 8
Design of the Stereo system
• Background simplification
• Camera calibration – essential step
• Tracking algorithm
•
•
•
•
Top 1/3 of the image
Instrument identification
Image de-warping
Icon tracking
• Complications
• Multiple instruments –
• Occlusion
• Correspondence disambiguation
Haptica ProMIS May2006
Page 9
Vision Systems Design
• Off the shelf web cams (firewire)
• Calibration of cameras to sub pixel
accuracy
• High frame rate (15fps or higher)
• Task trays manufactured to high accuracy
• Task trays rigidly mounted to high
accuracy
• Key design factor in enabling low cost
Augmented Reality
Bile duct
AR
Haptica ProMIS May2006
Page 10
The Augmented Reality system
• Aligning parallel 3D Universe’s
• Vision System reference frame
• Graphics Engine reference frame
• Mechanical mounting reference frame
• Registration
• Optical Distortion
• De-warping of images
• Occlusion masks (instruments and objects)
• Real-time tracking and AR display (on a laptop)
• down and dirty in direct x / direct show
Haptica ProMIS May2006
Page 11
Augmented Reality Concept
• Annotate live video with Computer Mediated
Learning content
• Teaching Instructions
• Feedback on Progress
• Hybrid tasks
• Adverse Events
AR
Demo
Haptica ProMIS May2006
Page 12
Results
• ProMIS Construct Validity Trials
• RCSI / AMNCH
• Emory University
• Imperial College
• Very high ability to discriminate novice from expert
using economy of movement
• Papers published in surgical fora
• Other Validations
• European IST prize
• Irish Software Association Technical Innovation Prize
• Society of Laparoendoscopic Surgeons Innovation of the Year
award
• Irish Design Institute “Highly Commended Design”
Haptica ProMIS May2006
Page 13
Idea refinement – The importance of key reference customers
• Concept to product
• March 02 to December 03
• Patent May 02
• Basic concept – cardboard box
• Stage Gate™ type process
• Multiple Prototypes
• with RCSI, Emory (US), Imperial (UK)
• Production prototypes June 03
• Production process set-up Oct 03
• First sales to RCSI December 03
Haptica ProMIS May2006
Page 14
Current Position
• ProMIS 2 Launched October 2005
• Hand Tracking for “hand assisted” Procedures
• Model abdominal organs
• Full procedure training for Colectomy
• 8 cameras, tilt-sensor, animatronics
• Still runs on a laptop…
• 7 Papers validating ProMIS published at most
recent surgical congress
Haptica ProMIS May2006
Page 15
ProMIS 2.0
Haptica ProMIS May2006
Page 16
Haptica ProMIS May2006
Page 17
Haptica ProMIS May2006
Page 18
Haptica ProMIS May2006
Page 19
Haptica ProMIS May2006
Page 20
Haptica ProMIS May2006
Page 21
Future of Augmented Reality and Video Indexation
• Current Projects
• Illicit Content detection
• Colonoscopy: Automated Reporting tools - tagging
• Colonoscopy: Automated cancer detection
• Hand washing quality assessment
• Ground Collision Warning System for Aircraft
Haptica ProMIS May2006
Page 22