Transcript A Simple Introduction to Support Vector Machines

Vectors..
Vectors: notations

x2
B2
A2
A vector in a n-dimensional space in described by a n-uple of real
numbers
 A1 
A   2 
A 
 B1 
B   2 
B 
A
A1

A  A
T

1
BT  B1
A

B2

2
B
1
B1 x
2
Vectors: sum

The components of the sum vector are the sums of the components
C  A B
 C 1   A1  B1 
 2 2

C   A  B2 
  

x2
C2
C
B2
A2
A
A1
B
B1 C1
x1
3
Vectors: difference

The components of the sum vector are the sums of the components
C  B A
 C 1   B1  A1 
 2 2

 C   B  A2 
  

x2
B2
C2
A2
A
A1
B
C
C1 B 1
x1
-A
4
Vectors: product by a scalar

The components of the sum vector are the difference of the
components
C  a A
 C 1   a  A1 
 2

 C   a  A2 
  

x2
C2
A2
3A
A
A1
C1
x1
5
Vectors: Norm

The most simple definition for a norm is the euclidean module of the
components
A 
 A 
i 2
1. || x  y |||| x ||  || y ||
2. || x ||  || x ||
3. || x || 0 se x  0
i
x2
A2
A
A 
A1
A   A 
1 2
2 2
x1
6
Vectors: distance between two points

The distance between two points is the norm of the difference vector
d  A, B  A  B  B  A
x2
B2
C2
A2
A
A1
B
C
C1 B 1
d  A, B  
B
1
  B
1 2
A
2
A

2 2
x1
-A
7
Vectors: Scalar product

The components of the sum vector are the sums of the components
c  AB  AT  B   Ai Bi
x2
1.
x, y  y, x
2.
x  y,z  x, z  y, z
3.
x, y   x, y
4.
x, x  0
B2
A2
i
x, y  z  x, y  x, z
e x, y   x, y
B
A
θ
A1
e
B1
x1
c  A  B  cos
8
Vectors: Scalar product
v
v


u
  90
u
  90
v, u  0
v, u  0
v

u
  90
v, u  0
9
Vectors: Norm and scalar product

The components of the sum vector are the sums of the components
A 
 A 
i 2
 AT  A 
A, A
i
10
Vectors: Definition of an hyperplane
In R2 , an hyperplane is a line
A line passing through the origin can be defined with as the set of the
vectors that are perpendicular to a given vector W
x2
XW  W T X  0
W
W 1X 1 W 2 X 2  0
x1
11
Vectors: Definition of an hyperplane
In R3 , an hyperplane is a plane
A plane passing through the origin can be defined with as the set of
the vectors that are perpendicular to a given vector W
x3
XW  W T X  0
W 1X 1 W 2 X 2 W 3 X 3  0
W
x1
x2
12
Vectors: Definition of an hyperplane
In R2 , an hyperplane is a line
A line perpendicular to W and whose distance from the origin is equal
to b is defined by the points whose scalar vector with W is equal to b
XW
x2
W
X
-b/|W|
W
x1
WT X b


W
W
W X W X  b  0
1
1
2
2
-b>0
13
Vectors: Definition of an hyperplane
In R2 , an hyperplane is a line
A line perpendicular to W and whose distance from the origin is equal
to b is defined by the points whose scalar vector with W is equal to b
XW
x2
W
WT X b


W
W
X
W
b/||W||
x1
W X W X  b  0
1
1
2
2
-b<0
14
Vectors: Definition of an hyperplane
In Rn , an hyperplane is defined by
XW  b  W X  b  0
T
15
An hyperplane divides the space
A
<AW>/||W||
x2
X
<BW>/||W||
-b/||W||
W
x1
T
AW  W A  b
T
BW  W B  b
B
16
Distance between a hyperplane and a point
A
<AW>/||W||
x2
X
<BW>/||W||
-b/||W||
W
d ( A, r ) 
x1
d ( B, r ) 
B
AW  b
W
BW  b
W
17
Distance between two parallel hyperplane
W X  b' 0
T
W X b  0
T
x2
-b’/||W||
d (r , r ' ) 
W
-b/||W||
x1
b  b'
W
18
Lagrange Multipliers
Aim
We want to maximise the function z = f(x,y)
subject to the constraints
g(x,y) = c (curve in
the x,y plane)
7/18/2015
20
Simple solution
Solve the constraint g(x,y) = c and express, for
example, y=h(x)
The substitute in function f and find the maximum
in x of
f(x, h(x))
Analytical solution of the constraint can be very
difficult
7/18/2015
21
Geometrical interpretation
The level contours of f(x,y) are defined by f(x,y) = dn
22
Lagrange Multipliers
Suppose we walk along the contour line with g = c.
In general the contour lines of f and g may be distinct:
traversing the contour line for g = c we cross the contour lines
of f.
While moving along the contour line for g = c the value of f
can vary.
Only when the contour line for g = c touches contour lines of f
tangentially, we do not increase or decrease the value of f that is, when the contour lines touch but do not cross.
23
Normal to a curve
24
Gradient of a curve
Given a curve g(x,y) = c
the gradient of g is:
(x,y)
(x+εx, x+εy)
 g g 
g   , 
 x y 
Consider 2 points of the curve: (x,y); (x+εx, x+εy),
for small ε
g
g x   x , y   y   g x, y    x
x
g
x
y
( x, y )

( x, y )
T


 g x, y  ε g ( x , y )
25
Gradient of a curve
Given a curve g(x,y) = c
the gradient of g is:
(x,y)
(x+εx, x+εy)
Since both points satisfy the curve
T
equation:
c  c  ε g ( x , y )
ε
grad (g)
εT g ( x , y )  0
For small ε, ε is parallel to the curve and,
consequently, the gradient is perpendicular to the
curve
26
Lagrange Multipliers
The point on g(x,y)=c that
Max-min-imize f(x,y) the gradient
of f is perpendicular to the curve
g, otherwise we should increase or decrease f by moving
locally on the curve
So, the two gradients are parallel
for some scalar λ (where  is the gradient).
27
Lagrange Multipliers
Thus we want points (x,y) where g(x,y) = c and
,
To incorporate these conditions into one equation, we introduce
an auxiliary function (Lagrangian)
F ( x, y ,  )  f ( x, y )    g ( x, y )  c 
and solve
.
28
Recap of Constrained Optimization
Suppose we want to: minimize/maximize f(x) subject to
g(x) = 0
 A necessary condition for x0 to be a solution:

a: the Lagrange multiplier
 For multiple constraints gi(x) = 0, i=1, …, m, we need a
Lagrange multiplier ai for each of the constraints

-
29
Constrained Optimization: inequality
We want to maximize f(x,y) with inequality constraint
g(x,y)c.
 The search must be confined in the red portion
(gradient of a function points towards the direction along
which it increases)
g(x,y) ≤ c

Constrained Optimization: inequality
maximize f(x,y) with inequality constraint g(x,y)c.
 If the gradients are opposite (<0) the function
increases in the allowed portion The maximum
cannot be on the curve g(xy)=c
 Maximum is on the curve only if >0

g(x,y) ≤ c
f increases,
0
F ( x, y ,  )  f ( x, y )    g ( x, y )  c 
Constrained Optimization: inequality
Minimize f(x,y) with inequality constraint g(x,y)c.
 If the gradients are opposite (<0) the function
increases in the allowed portion
 Minimum is on the curve only if <0

g(x,y) ≤ c
f increases,
0
F ( x, y ,  )  f ( x, y )    g ( x, y )  c 
Constrained Optimization: inequality
maximize f(x,y) with inequality constraint g(x,y)≥c.
 If the gradients are opposite (<0) the function
decreases in the allowed portion
 Maximum is on the curve only if <0

F ( x, y ,  )  f ( x, y )    g ( x, y )  c 
g(x,y) ≥ c
0
f decreases,
Constrained Optimization: inequality
Minimize f(x,y) with inequality constraint g(x,y)≥c.
 If the gradients are opposite (<0) the function
decreases in the allowed portion
 Minimum is on the curve only if >0

F ( x, y ,  )  f ( x, y )    g ( x, y )  c 
g(x,y) ≥ c
0
f decreases,
Karush-Kuhn-Tucker conditions
The function f(x) subject to constraints gi(x) ≤or≥ 0 is
max-minimized by opimizing the Lagrange function
F ( x, a i )  f ( x)  a i  gi ( x)
i
with αi satisfying the following conditions:
and
MIN
gi(x) ≤ 0
αi ≥ 0
gi(x) ≥ 0
αi ≤ 0
MAX
αi ≤ 0
αi ≥ 0
a i  g i ( x0 )  0, i
35
Constrained Optimization: inequality

Karush-Kuhn-Tucker complementarity condition
a i  g i ( x0 )  0, i
means that
a i  0  gi ( xo )  0
The constraint is active only on the border, and cancel out
in the internal regions
36
Concave-Convex functions
Concave
Convex
7/18/2015
37
Dual problem

If f(x) is a convex function
Is solved by:
From the first equation we can find x as a function of the ai
These can be substituted in the Lagrangian function
obtaining the dual Lagrangian function

L(a i )  inf L( x, a i )  inf  f ( x) 
x
x 


ai gi ( x) 
i

38
Dual problem

L(a i )  inf L( x, a i )  inf  f ( x) 
x
x 




ai gi ( x) 
i

The dual Lagrangian is concave: maximising it with
respect to ai ,with ai>0, solve the original constrained
problem. We compute ai as:





max L(a i )  max inf L( x, a i )  max inf f ( x)  i ai gi ( x)
ai
ai
x
ai
x
Then we can obtain x by substituting using the
expression of x as a function of ai
39

Dual problem:trivial example
Minimize the function f(x)=x2 with the constraint x≤-1
(trivial: x=-1)

The Lagrangian is
L( x, a )  x 2  a ( x  1)
Minimising with respect to x
-1
L
 0  2x  a  0  x  a
2
x
The dual Lagrangian is
Maximising it gives: a=2
Then subsituting,
L(a ) 
a2
4

a2
2
a  a 
a2
4
x  a  1
2
40
An Introduction to Support Vector
Machines
What is a good Decision Boundary?
Consider a two-class, linearly
separable classification problem
 Many decision boundaries!



Class 2
The Perceptron algorithm can be
used to find such a boundary
Are all decision boundaries
equally good?
Class 1
42
Examples of Bad Decision Boundaries
Class 2
Class 1
Class 2
Class 1
43
Large-margin Decision Boundary

The decision boundary should be as far away from the
data of both classes as possible
 We should maximize the margin, m
Class 2
Class 1
m
44
Hyperplane Classifiers(2)
w  xi  b  1 for yi  1
w  xi  b  1 for yi  1
45
Finding the Decision Boundary

Let {x1, ..., xn} be our data set and let yi  {1,-1} be
the class label of xi
For yi=1
y=1
For yi=-1
y=1
So:
y=1
y=-1
y=1
y=1
y=-1

w xi  b  1
T
w xi  b  1
T

yi  w xi  b  1, xi , yi 
T
Class 2
y=-1
y=-1
y=-1
Class 1
y=-1
m
46
Finding the Decision Boundary



The decision boundary should classify all points correctly

The decision boundary can be found by solving the
following constrained optimization problem
This is a constrained optimization problem. Solving it
requires to use Lagrange multipliers
47
Finding the Decision Boundary

The Lagrangian is
ai≥0
2
T
 Note that ||w|| = w w

48
Gradient with respect to w and b

Setting the gradient of
have

w.r.t. w and b to zero, we


n
1 T
L  w w   a i 1  yi wT xi  b 
2
i 1
1 m k k n 
 m k k

  w w   a i 1  yi   w xi  b  
2 k 1
i 1
 k 1


n: no of examples, m: dimension of the space
 L
 wk  0, k

 L
 b  0


49
The Dual Problem

If we substitute
to
, we have
Since

This is a function of ai only
50
The Dual Problem
The new objective function is in terms of ai only
 It is known as the dual problem: if we know w, we
know all ai; if we know all ai, we know w
 The original problem is known as the primal problem
 The objective function of the dual problem needs to be
maximized (comes out from the KKT theory)
 The dual problem is therefore:

Properties of ai when we introduce
the Lagrange multipliers
The result when we differentiate the
original Lagrangian w.r.t. b
51
The Dual Problem

This is a quadratic programming (QP) problem


A global maximum of ai can always be found
w can be recovered by
52
Characteristics of the Solution

Many of the ai are zero
w is a linear combination of a small number of data points
 This “sparse” representation can be viewed as data
compression as in the construction of knn classifier


xi with non-zero ai are called support vectors (SV)
 The decision boundary is determined only by the SV
 Let tj (j=1, ..., s) be the indices of the s support vectors.
We can write

Note: w need not be formed explicitly
53
A Geometrical Interpretation
Class 2
a8=0.6 a10=0
a7=0
a5=0
a4=0
a9=0
Class 1
a2=0
a1=0.8
a6=1.4
a3=0
54
Characteristics of the Solution

For testing with a new data z


Compute
and
classify z as class 1 if the sum is positive, and class 2
otherwise
Note: w need not be formed explicitly
55
The Quadratic Programming Problem

Many approaches have been proposed


Loqo, cplex, etc. (see http://www.numerical.rl.ac.uk/qp/qp.html)
Most are “interior-point” methods
Start with an initial solution that can violate the constraints
 Improve this solution by optimizing the objective function
and/or reducing the amount of constraint violation


For SVM, sequential minimal optimization (SMO) seems
to be the most popular
A QP with two variables is trivial to solve
 Each iteration of SMO picks a pair of (ai,aj) and solve the
QP with these two variables; repeat until convergence


In practice, we can just regard the QP solver as a
“black-box” without bothering how it works
56
Non-linearly Separable Problems
We allow “error” xi in classification; it is based on the
output of the discriminant function wTx+b
 xi approximates the number of misclassified samples

Class 2
Class 1
57
Soft Margin Hyperplane

The new conditions become
xi are “slack variables” in optimization
 Note that xi=0 if there is no error for xi
 xi is an upper bound of the number of errors


We want to minimize

C : tradeoff parameter between error and margin
n
1 2
w  C  xi
2
i 1
58
The Optimization Problem



n
n
n
1 T
L  w w  C  xi a i 1  xi  yi wT xi  b   ixi
2
i 1
i 1
i 1
With α and μ Lagrange multipliers, POSITIVE
n
L
 w j  a i yi xij  0
w j
i 1
 n

w   a i yi xi  0
i 1
L
 C a j   j  0
x j
L n
  yia i  0
b i 1
59
The Dual Problem
n
T
1 n n
L   a ia j yi y j xi x j  C  x i 
2 i 1 j 1
i 1
n


 n

T


  a i 1  xi  yi   a j y j x j xi  b     ix i

 i 1
i 1
j

1



n
With
n
 ya
i 1
i
i
C aj  j
0
T
1
L   a ia j yi y j xi x j  a i
2 i 1 j 1
i 1
n
n
n
The Optimization Problem

The dual of this new constrained optimization problem is
New constrainsderive from C  a j
are positive.
 w is recovered as

  j since μ and α
This is very similar to the optimization problem in the
linear separable case, except that there is an upper
bound C on ai now
 Once again, a QP solver can be used to find ai

61
n
1 2
w  C  xi
2
i 1
The algorithm try to keep ξ null, maximising the
margin
 The algorithm does not minimise the number of
error. Instead, it minimises the sum of distances
fron the hyperplane


When C increases the number of errors tend to
lower. At the limit of C tending to infinite, the
solution tend to that given by the hard margin
formulation, with 0 errors
7/18/2015
62
Soft margin is more robust
63
Extension to Non-linear Decision Boundary
So far, we have only considered large-margin classifier
with a linear decision boundary
 How to generalize it to become nonlinear?
 Key idea: transform xi to a higher dimensional space to
“make life easier”

Input space: the space the point xi are located
 Feature space: the space of f(xi) after transformation


Why transform?
Linear operation in the feature space is equivalent to nonlinear operation in input space
 Classification can become easier with a proper
transformation. In the XOR problem, for example, adding a
new feature of x1x2 make the problem linearly separable

64
XOR
Is not linearly separable
X
0
Y
0
0
0
1
1
1
0
1
1
1
0
Is linearly separable
X
0
Y
0
XY
0
0
0
1
1
1
0
1
0
0
1
1
1
0
65
Find a feature space
S.Mika: Kernel Fisher Discriminant
66
Transforming the Data
f(.)
Input space
f( )
f( )
f( )
f( ) f( ) f( )
f( )
f( )
f( )
f( ) f( )
f( ) f( )
f( )
f( ) f( )
f( )
f( )
Feature space
Note: feature space is of higher dimension
than the input space in practice

Computation in the feature space can be costly because it is
high dimensional


The feature space is typically infinite-dimensional!
The kernel trick comes to rescue
67
Transforming the Data
f(.)
Input space
f( )
f( )
f( )
f( ) f( ) f( )
f( )
f( )
f( )
f( ) f( )
f( ) f( )
f( )
f( ) f( )
f( )
f( )
Feature space
Note: feature space is of higher dimension
than the input space in practice

Computation in the feature space can be costly because it is
high dimensional


The feature space is typically infinite-dimensional!
The kernel trick comes to rescue
68
The Kernel Trick

Recall the SVM optimization problem
The data points only appear as inner product
 As long as we can calculate the inner product in the
feature space, we do not need the mapping explicitly
 Many common geometric operations (angles, distances)
can be expressed by inner products
 Define the kernel function K by

69
An Example for f(.) and K(.,.)

Suppose f(.) is given as follows

An inner product in the feature space is


So, if we define the kernel function as follows, there is
no need to carry out f(.) explicitly
This use of kernel function to avoid carrying out f(.)
explicitly is known as the kernel trick
70
Kernels

Given a mapping:
x  φ(x)
a kernel is represented as the inner product
K (x, y) 
 φ (x)φ (y)
i
i
i
A kernel must satisfy the Mercer’s condition:

g (x) such that g 2 (x)dx  0 

K (x,y ) g (x) g (y )dxdy  0
71
Modification Due to Kernel Function
Change all inner products to kernel functions
 For training,

Original
With kernel
function
72
Modification Due to Kernel Function

For testing, the new data z is classified as class 1 if f 0,
and as class 2 if f <0
Original
With kernel
function
73
More on Kernel Functions

Since the training of SVM only requires the value of K(xi,
xj), there is no restriction of the form of xi and xj



xi can be a sequence or a tree, instead of a feature vector
K(xi, xj) is just a similarity measure comparing xi and xj
For a test object z, the discriminat function essentially is
a weighted sum of the similarity between z and a preselected set of objects (the support vectors)
74
Example

Suppose we have 5 1D data points

x1=1, x2=2, x3=4, x4=5, x5=6, with 1, 2, 6 as class 1 and 4,
5 as class 2  y1=1, y2=1, y3=-1, y4=-1, y5=1
75
Example
class 1
class 1
class 2
1
2
4
5
6
76
Example

We use the polynomial kernel of degree 2
K(x,y) = (xy+1)2
 C is set to 100


We first find ai (i=1, …, 5) by
77
Example

By using a QP solver, we get
a1=0, a2=2.5, a3=0, a4=7.333, a5=4.833
 Note that the constraints are indeed satisfied
 The support vectors are {x2=2, x4=5, x5=6}


The discriminant function is

b is recovered by solving f(2)=1 or by f(5)=-1 or by f(6)=1,

All three give b=9
78
Example
Value of discriminant function
class 1
class 1
class 2
1
2
4
5
6
79
Kernel Functions
In practical use of SVM, the user specifies the kernel
function; the transformation f(.) is not explicitly stated
 Given a kernel function K(xi, xj), the transformation f(.)
is given by its eigenfunctions (a concept in functional
analysis)

Eigenfunctions can be difficult to construct explicitly
 This is why people only specify the kernel function without
worrying about the exact transformation


Another view: kernel function, being an inner product, is
really a similarity measure between the objects
80
A kernel is associated to a transformation


Given a kernel, in principle it should be recovered the
transformation in the feature space that originates it.
K(x,y) = (xy+1)2= x2y2+2xy+1
It corresponds the transformation
 x 


x   2x 
 1 


2
7/18/2015
81
Examples of Kernel Functions

Polynomial kernel up to degree d

Polynomial kernel up to degree d

Radial basis function kernel with width s


The feature space is infinite-dimensional
Sigmoid with parameter k and q

It does not satisfy the Mercer condition on all k and q
82
Example
83
Building new kernels

If k1(x,y) and k2(x,y) are two valid kernels then the
following kernels are valid
 Linear Combination
k ( x, y)  c1k1 ( x, y)  c2 k2 ( x, y)

Exponential

Product

k ( x, y)  expk1 ( x, y)
k ( x, y)  k1 ( x, y)  k2 ( x, y)
Polymomial tranfsormation (Q: polymonial with non negative
coeffients)
k ( x, y)  Qk1 ( x, y)

Function product (f: any function)
k ( x, y)  f ( x)k1 ( x, y) f ( y)
84
Ploynomial kernel
Ben-Hur et al, PLOS computational Biology 4 (2008)
85
Gaussian RBF kernel
Ben-Hur et al, PLOS computational Biology 4 (2008)
86
Spectral kernel for sequences

Given a DNA sequence x we can count the number of
bases (4-D feature space)
f1 ( x)  (nA , nC , nG , nT )

Or the number of dimers (16-D space)
f2 ( x)  (nAA , nAC , nAG , nAT , nCA , nCC , nCG , nCT ,..)

Or l-mers (4l –D space)

The spectral kernel is
kl ( x, y)  fl xfl  y 
7/18/2015
87
Choosing the Kernel Function
Probably the most tricky part of using SVM.
 The kernel function is important because it creates the
kernel matrix, which summarizes all the data
 Many principles have been proposed (diffusion kernel,
Fisher kernel, string kernel, …)
 There is even research to estimate the kernel matrix
from available information

In practice, a low degree polynomial kernel or RBF
kernel with a reasonable width is a good initial try
 Note that SVM with RBF kernel is closely related to RBF
neural networks, with the centers of the radial basis
functions automatically chosen for SVM

88
Why SVM Work?
The feature space is often very high dimensional. Why
don’t we have the curse of dimensionality?
 A classifier in a high-dimensional space has many
parameters and is hard to estimate
 Vapnik argues that the fundamental problem is not the
number of parameters to be estimated. Rather, the
problem is about the flexibility of a classifier
 Typically, a classifier with many parameters is very
flexible, but there are also exceptions

Let xi=10i where i ranges from 1 to n. The classifier
can classify all xi correctly for all possible
combination of class labels on xi
 This 1-parameter classifier is very flexible

89
Why SVM works?

Vapnik argues that the flexibility of a classifier should
not be characterized by the number of parameters, but
by the flexibility (capacity) of a classifier

This is formalized by the “VC-dimension” of a classifier
Consider a linear classifier in two-dimensional space
 If we have three training data points, no matter how
those points are labeled, we can classify them perfectly

90
VC-dimension

However, if we have four points, we can find a labeling
such that the linear classifier fails to be perfect
We can see that 3 is the critical number
 The VC-dimension of a linear classifier in a 2D space is 3
because, if we have 3 points in the training set, perfect
classification is always possible irrespective of the
labeling, whereas for 4 points, perfect classification can
be impossible

91
VC-dimension
The VC-dimension of the nearest neighbor classifier is
infinity, because no matter how many points you have,
you get perfect classification on training data
 The higher the VC-dimension, the more flexible a
classifier is
 VC-dimension, however, is a theoretical concept; the VCdimension of most classifiers, in practice, is difficult to
be computed exactly


Qualitatively, if we think a classifier is flexible, it probably
has a high VC-dimension
92
Other Aspects of SVM

How to use SVM for multi-class classification?
One can change the QP formulation to become multi-class
 More often, multiple binary classifiers are combined




One can train multiple one-versus-all classifiers, or combine
multiple pairwise classifiers “intelligently”
How to interpret the SVM discriminant function value as
probability?


See DHS 5.2.2 for some discussion
By performing logistic regression on the SVM output of a
set of data (validation set) that is not used for training
Some SVM software (like libsvm) have these features
built-in
93
Software
A list of SVM implementation can be found at
http://www.kernel-machines.org/software.html
 Some implementation (such as LIBSVM) can handle
multi-class classification
 SVMLight is among one of the earliest implementation of
SVM
 Several Matlab toolboxes for SVM are also available

94
Summary: Steps for Classification
Prepare the pattern matrix
 Select the kernel function to use
 Select the parameter of the kernel function and the
value of C


You can use the values suggested by the SVM software, or
you can set apart a validation set to determine the values
of the parameter
Execute the training algorithm and obtain the ai
 Unseen data can be classified using the ai and the
support vectors

95
Strengths and Weaknesses of SVM

Strengths

Training is relatively easy

No local optimal, unlike in neural networks
It scales relatively well to high dimensional data
 Tradeoff between classifier complexity and error can be
controlled explicitly
 Non-traditional data like strings and trees can be used as
input to SVM, instead of feature vectors


Weaknesses

Need to choose a “good” kernel function.
96
Other Types of Kernel Methods
A lesson learnt in SVM: a linear algorithm in the feature
space is equivalent to a non-linear algorithm in the input
space
 Standard linear algorithms can be generalized to its nonlinear version by going to the feature space


Kernel principal component analysis, kernel independent
component analysis, kernel canonical correlation analysis,
kernel k-means, 1-class SVM are some examples
97
Conclusion
SVM is a useful alternative to neural networks
 Two key concepts of SVM: maximize the margin and the
kernel trick
 Many SVM implementations are available on the web for
you to try on your data set!

98
Resources
http://www.kernel-machines.org/
 http://www.support-vector.net/
 http://www.support-vector.net/icml-tutorial.pdf
 http://www.kernel-machines.org/papers/tutorialnips.ps.gz
 http://www.clopinet.com/isabelle/Projects/SVM/applist.h
tml

99
SVM-light
http://svmlight.joachims.org
 Author: Thorsten Joachims , Cornell University





Can be downloaded and easily installed
http://download.joachims.org/svm_light/current/svm_light.tar.gz
To install SVMlight you need to download svm_light.tar.gz. Create a new
directory: mkdir svm_light
Move svm_light.tar.gz to this directory and unpack it with
gunzip -c svm_light.tar.gz | tar xvf  Now execute make or make all
Two programs are compiled:
svm_learn (learning module)
svm_classify (classification module)
100
SVM-light: Training Input
1 1:2 2:1 3:4 4:3
1 1:2 2:1 3:4 4:3
-1 1:2 2:1 3:3 4:0
1 1:2 2:2 3:3 4:3
1 1:2 2:4 3:3 4:2
-1 1:2 2:2 3:3 4:0
-1 1:2 2:0 3:3
-1 1:2 2:4 3:3
-1 1:4 2:5 3:3
1 1:2 2:2 3:3 4:2
Class FeatureN:ValueN
101
SVM-light: Training
svm_learn [options] example_file model_file
SOME OPTIONS
General options:
-? - this help
Learning options:
-c float: trade-off between training error and margin (default [avg. x*x]^-1)
Performance estimation options:
-x [0,1] - compute leave-one-out estimates (default 0
Kernel options:
-t int - type of kernel function: 0: linear (default) 1: polynomial (s a*b+c)^d 2: radial
basis function exp(-gamma ||a-b||^2) 3: sigmoid tanh(s a*b + c) 4: user defined
kernel from kernel.h
-d int - parameter d in polynomial kernel
-g float - parameter gamma in rbf kernel
-s float - parameter s in sigmoid/poly kernel -r float - parameter c in sigmoid/poly
kernel
-u string - parameter of user defined kernel Optimization
102
SVM-light: Trained Model
SVM-light Version V6.02
0 # kernel type
3 # kernel parameter -d
1 # kernel parameter -g
1 # kernel parameter -s
1 # kernel parameter -r
empty# kernel parameter -u
4 # highest feature index
12 # number of training documents
13 # number of support vectors plus 1
1.0380931 # threshold b, each following line is a SV (starting with alpha*y)
0.03980964156284725469214791360173 1:2 2:4 3:3 4:0 #
-0.018316632908270628204983054843069 1:4 2:5 3:3 4:0 #
-0.03980964156284725469214791360173 1:2 2:1 3:3 4:0 #
0.03980964156284725469214791360173 1:2 2:1 3:4 4:3 #
-0.03980964156284725469214791360173 1:2 2:0 3:3 4:0 #
0.03980964156284725469214791360173 1:2 2:4 3:3 4:2 #
0.03980964156284725469214791360173 1:2 2:2 3:3 4:2 #
0.03980964156284725469214791360173 1:2 2:1 3:4 4:3 #
-0.037841055392657176048576417315417 1:3 2:1 3:3 4:0 #
-0.03980964156284725469214791360173 1:2 2:2 3:3 4:0 #
0.03980964156284725469214791360173 1:2 2:2 3:3 4:3 #
0.016345916179801231460366750525282 1:1 2:2 3:4 4:3 #
103
SVM-light: Predicting

svm_classify [options] example_file model_file output_file
104
SVM-light: Prediction
0.88647894
0.88647894
0.81321667
0.24665358
0.29204665
0.99999997
-0.8864726
-0.93186567
-0.84107953
-1.000032
-0.90916914
-0.99999364
105