Transcript Phase recognition out of current images

Chair for Computer Aided Medical Procedures & Augmented Reality
|
wwwnavab.cs.tum.edu
Automatic Feature Generation for
Endoscopic Image Classification
Ulrich Klank
18 July 2015
Chair for Computer Aided Medical Procedures & Augmented Reality
Department of Computer Science | Technische Universität München
Chair for Computer Aided Medical Procedures & Augmented Reality
|
wwwnavab.cs.tum.edu
Endoscopic Image Classification
 Synchronization of surgeries by
phases is essential for Workflow
Project1
 Analyzing endoscopic images
supports the classification of the
current state into a phase.
1 A. Ahmadi, T. Sielhorst, R. Stauder, M. Horn, H. Feussner, N. Navab -Recovery of surgical workflow without explicit models
CAMP | Department of Computer Science | Technische Universität München | 18 July 2015
2
Chair for Computer Aided Medical Procedures & Augmented Reality
|
wwwnavab.cs.tum.edu
Task
 How can these images be
classified into surgical phases?
 Or if they can’t be classified well
enough, which features can be
added to the rest of the
information?
CAMP | Department of Computer Science | Technische Universität München | 18 July 2015
3
Chair for Computer Aided Medical Procedures & Augmented Reality
|
wwwnavab.cs.tum.edu
Definition of a Feature
Gradient Histogram
 Feature as representation of discriminative
information
 Numerical vector calculated out of the
image
 Reduction of the dimension
 Features
 Basic: Gradient/Color/Motion Histogram
 Integral image1
 Phase concruency2
1 Viola, Jones- Robust Real-time Object Detection
2
Kovesi - Image Features from Phase Congruency
CAMP | Department of Computer Science | Technische Universität München | 18 July 2015
4
Chair for Computer Aided Medical Procedures & Augmented Reality
|
wwwnavab.cs.tum.edu
Feature Space

With a limit to algorithms with an output smaller or equal than the input
we get a number of injective function about:
(cd)
cd ( cd ¡ 1)
2
depending on the dimension of the input vector d and the number of
different values c which are possible in the input.

Exploration of this space with Genetic Programming1
 A method to explore a high dimensional space
 Fitness function influences the resulting features
1 Feature Extraction from Multiple Data Sources Using Genetic Programming - John J. Szymanski,* Steven P. Brumby, Paul Pope,
Damian Eads, Diana Esch-Mosher,Mark Galassi, Neal R. Harvey, Hersey D.W. McCulloch, Simon J. Perkins, Reid Porter,James Theiler,
A. Cody Young, Jeffrey J. Bloch and Nancy David
CAMP | Department of Computer Science | Technische Universität München | 18 July 2015
5
Chair for Computer Aided Medical Procedures & Augmented Reality
|
wwwnavab.cs.tum.edu
Feature Generation by Genetic Programming12
Code
Evaluation
Mutation
1 Wolfgang Banzhaf, Peter Nordin, Robert E. Keller, Frank D. Francone - Genetic Programming: An Introduction
2
Koza - Genetic Programming
CAMP | Department of Computer Science | Technische Universität München | 18 July 2015
6
Chair for Computer Aided Medical Procedures & Augmented Reality
|
wwwnavab.cs.tum.edu
Implementation Steps
 Definition of a programming language (more)
 Designed for recombination and mutation
 Designing a Framework to run programs
 Translation of basic features in the new language
 Evaluation of the fitness of programs
 Recombination and Mutation
CAMP | Department of Computer Science | Technische Universität München | 18 July 2015
7
Chair for Computer Aided Medical Procedures & Augmented Reality
|
wwwnavab.cs.tum.edu
An Example as Proof of Concept
 Implementation of a color histogram in the new
language
 Test on an easy sample, 6 images(10*10*RGB, jpeg)
in 3 classes:
 Dimension of input here (cd): 76.800
CAMP | Department of Computer Science | Technische Universität München | 18 July 2015
8
Chair for Computer Aided Medical Procedures & Augmented Reality
|
wwwnavab.cs.tum.edu
Evaluation of the Example
 The Fitness Function is the core of the
system
CAMP | Department of Computer Science | Technische Universität München | 18 July 2015
9
Chair for Computer Aided Medical Procedures & Augmented Reality
|
wwwnavab.cs.tum.edu
Evaluation of the Example
Comparison of several Programs Pi
P1
CAMP | Department of Computer Science | Technische Universität München | 18 July 2015
P2
…
10
Chair for Computer Aided Medical Procedures & Augmented Reality
|
wwwnavab.cs.tum.edu
Evaluation of the Example
P1
P2
…
A random selection of the labeled input Ik
several from of each class
I1
CAMP | Department of Computer Science | Technische Universität München | 18 July 2015
I2
I3
11
Chair for Computer Aided Medical Procedures & Augmented Reality
|
wwwnavab.cs.tum.edu
Evaluation of the Example
P1
I1
…
P2
I2
I3
Execution of the program with every
selected input Ik
As result there are output vectors:
o11 ; o12 ¢¢¢o23
CAMP | Department of Computer Science | Technische Universität München | 18 July 2015
12
Chair for Computer Aided Medical Procedures & Augmented Reality
|
wwwnavab.cs.tum.edu
Evaluation of the Example
P1
I1
…
P2
I2
I3
o11 ; o12 ¢¢¢o23
Compare each pair of output vectors of one
Program, by the distance
di
k ;l
=
P
l en
( oi k [m ]¡ oi l [m ]) 2
m = 0 ( j o [m ]j + j o [m ]j ) 2
ik
il
CAMP | Department of Computer Science | Technische Universität München | 18 July 2015
13
Chair for Computer Aided Medical Procedures & Augmented Reality
|
wwwnavab.cs.tum.edu
Evaluation of the Example
P1
I1
…
P2
I2
I3
o11 ; o12 ¢¢¢o23
di
k;l
=
P
l en
(oi k [m ]¡ oi l [m ]) 2
m = 0 (jo [m ]j+ jo [m ]j) 2
ik
il
Select the maximum of these distances,
only considering pairs of different
classes:
ar gmax k l (di jk:class 6
= l:class)
kl
Now we have two classes k, l which our
feature separates
CAMP | Department of Computer Science | Technische Universität München | 18 July 2015
14
Chair for Computer Aided Medical Procedures & Augmented Reality
|
wwwnavab.cs.tum.edu
Evaluation of the example
P1
I1
…
P2
I2
I3
o11 ; o12 ¢¢¢o23
di
k;l
=
P
l en
(oi k [m ]¡ oi l [m ]) 2
m = 0 (jo [m ]j+ jo [m ]j) 2
ik
il
Subtract the distances in images of the
classes k or l
f i = di ¡
kl
di
+ di
kk
ll
2
ar gmax k l (di jk:class 6
= l:class)
kl
CAMP | Department of Computer Science | Technische Universität München | 18 July 2015
15
Chair for Computer Aided Medical Procedures & Augmented Reality
|
wwwnavab.cs.tum.edu
3.1 Results in this Example
< 64 >
< 72 >
< -16 >
New
Program
< 72 >
< -16 >
< -16 >
CAMP | Department of Computer Science | Technische Universität München | 18 July 2015
16
Chair for Computer Aided Medical Procedures & Augmented Reality
|
wwwnavab.cs.tum.edu
Fitness in the Example:
P1 
Input Program was calculating a Color Histogram (Output length
512 in ~2000 steps), resulting fitness: ~ 1.2 %
P2 
The new generated program calculated one number (in 30
steps) and got a fitness of: ~ 98.8 %
CAMP | Department of Computer Science | Technische Universität München | 18 July 2015
17
Chair for Computer Aided Medical Procedures & Augmented Reality
|
wwwnavab.cs.tum.edu
Evolution Technique used in the Example
 Selection of the two best Program and generation several
children by these two methods:
 One parental program: Mutation
P1
P2
 Adding of random commands
 Removing a single commands1
 Duplicating an existing command
P11 P1
2
 Switching commands
 Two parental programs: Breeding
 Mixing both, preferring blocks of a specific size
 Adding random commands in the gaps between
CAMP | Department of Computer Science | Technische Universität München | 18 July 2015
18
Chair for Computer Aided Medical Procedures & Augmented Reality
|
wwwnavab.cs.tum.edu
Conclusion
 The concept seems to work 
 Adaptation to workflow project’s data can start
CAMP | Department of Computer Science | Technische Universität München | 18 July 2015
19
Chair for Computer Aided Medical Procedures & Augmented Reality
|
wwwnavab.cs.tum.edu
Steps to adapt it to the Workflow project




Adapt fitness function (more)
Change Genetic rules (more)
Optimize the framework
Generate C code out of the intermediate
language
CAMP | Department of Computer Science | Technische Universität München | 18 July 2015
20
Chair for Computer Aided Medical Procedures & Augmented Reality
|
wwwnavab.cs.tum.edu
Any Questions?
CAMP | Department of Computer Science | Technische Universität München | 18 July 2015
21
Chair for Computer Aided Medical Procedures & Augmented Reality
|
wwwnavab.cs.tum.edu
Appendix
CAMP | Department of Computer Science | Technische Universität München | 18 July 2015
22
Chair for Computer Aided Medical Procedures & Augmented Reality
|
wwwnavab.cs.tum.edu
Planned Improvements in the Genetic rules
 Probability to survive should depend on the fitness of
all other existing programs
 Even a program with a low fitness should have a
small change to survive
 Also the reproduction should not be restricted to the
best program
Back
CAMP | Department of Computer Science | Technische Universität München | 18 July 2015
23
Chair for Computer Aided Medical Procedures & Augmented Reality
|
wwwnavab.cs.tum.edu
Fitness
 Actual fitness expresses:
 Linear separation of two classes
 Further improvements:
 Static Analysis of Code1
 Semantic Checks
 to prevent running in infinite loops
 to prevent the execution of programs not using the input
 …
 Other Classifiers to evaluate the output
Back
1
Colin G. Johnson-Deriving genetic programming fitness properties by static analysis
CAMP | Department of Computer Science | Technische Universität München | 18 July 2015
24
Chair for Computer Aided Medical Procedures & Augmented Reality
|
wwwnavab.cs.tum.edu
Specification of the language
 Assembler style with some high level functions
 1 Opcode and 3 Arguments for one command
 High level examples:
 Reading of the input and its dimensions
 For loops
 Adding of more functions possible
Back
CAMP | Department of Computer Science | Technische Universität München | 18 July 2015
25
Chair for Computer Aided Medical Procedures & Augmented Reality
|
wwwnavab.cs.tum.edu
Scheme for the programming language
OPCode
1. Parameter
2. Parameter
3. Parameter
010
0
100
65
For
From
To
End of the loop
counter to stack
Constant
Constant
Constant
123
-1
20
…
Read pixel
x
y
Value to stack
From stack
Constant
…
CAMP | Department of Computer Science | Technische Universität München | 18 July 2015
26
Chair for Computer Aided Medical Procedures & Augmented Reality
|
wwwnavab.cs.tum.edu
Generated Program (without nops)
INPUTDIMENSION 0 0 4595026
MULTIPLY 0 0 0
SAVE 0 -1 0
LOAD 3 0 0
INPUTDIMENSION 1 0 0
PLUS 0 12604 0
SAVE 23367 -1 0
FOR 0 -1 37
LOAD 4 0 0
PLUS 0 6457 0
PLUS 31190 0 0
LOAD 1 0 0
JUMP 55 0 0
SAVE 5 -1 0
DIVIDE 0 0 0
LOAD 5 0 0
DIVIDE 0 0 0
POP 0 0 -1
PUSH 32 0 0
READINPUT -1 -1 -1
LOAD 1 -1 0
LOAD 5 0 -1
POP 0 0 -1
PUSH 64 0 -1
LOAD 2 0 0
RETURN 0 0 0
SAVE 3 -1 0
JUMP 0 0 0
PUSH 8 0 0
FOR 8 520 58
CAMP | Department of Computer Science | Technische Universität München | 18 July 2015
27