Transcript Document 2985833

Beyond Linear Separability
Limitations of Perceptron
• Only linear separations
• Only converges for linearly separable data
One Solution (SVM’s)
Map data into a feature space where they are linearly separable
Another solution is
to use multiple
interconnected
perceptrons.
Artificial Neural Networks
• Interconnected networks of simple units
(let's call them "artificial neurons") in
which each connection has a weight.
– Weight wij is the weight of the ith input into
unit j.
– NN have some inputs where the feature
values are placed and they compute one or
more output values.
– Depending on the output value we determine
the class.
– If there are more than one output units we
choose the one with the greatest value.
• The learning takes place by adjusting the
weights in the network so that the desired
output is produced whenever a sample in
the input data set is presented.
Single Perceptron Unit
• We start by looking at a
simpler kind of "neural-like"
unit called a perceptron.
let x0  1 and w0  b
t hen t hehyperplaneequationis
wx  0
n
or
w x
j 0
j
j
0
h(x)  sign(w  x)
Depending on the value
of h(x) it outputs one
class or the other.
Beyond Linear Separability
• Since a single perceptron unit
can only define a single linear
boundary, it is limited to
solving linearly separable
problems.
• A problem like that illustrated
by the values of the XOR
boolean function cannot be
solved by a single perceptron
unit.
Multi-Layer Perceptron
• Solution: Combine multiple
linear separators.
• The introduction of "hidden"
units into NN make them
much more powerful:
– they are no longer limited to
linearly separable problems.
• Earlier layers transform the
problem into more tractable
problems for the latter layers.
Example: XOR problem
Output:
“class 0”
or
“class 1”
Example: XOR problem
Example: XOR problem
w23o2+w13o1+w03=0
w03=-1/2, w13=-1, w23=1
o2-o1-1/2=0
Multi-Layer Perceptron
• Any set of training points can be separated by a three-layer
perceptron network.
• “Almost any” set of points is separable by two-layer perceptron
network.
Autonomous Land Vehicle In a
Neural Network (ALVINN)
• ALVINN is an automatic steering system for a car based on
input from a camera mounted on the vehicle.
– Successfully demonstrated in a cross-country trip.
ALVINN
• The ALVINN neural
network is shown here. It
has
– 960 inputs (a 30x32
array derived from the
pixels of an image),
– 4 hidden units and
– 30 output units (each
representing a steering
command).
Optional material
Multi-Layer Perceptron Learning
• However, the presence of the discontinuous threshold in the
operation means that there is no simple local search for a good
set of weights;
– one is forced into trying possibilities in a combinatorial way.
• The limitations of the single-layer perceptron and the lack of a
good learning algorithm for multilayer perceptrons essentially
killed the field for quite a few years.
Soft Threshold
• A natural question to ask is whether we could use gradient
ascent/descent to train a multi-layer perceptron.
• The answer is that we can't as long as the output is discontinuous
with respect to changes in the inputs and the weights.
– In a perceptron unit it doesn't matter how far a point is from the
decision boundary, we will still get a 0 or a 1.
• We need a smooth output (as a function of changes in the
network weights) if we're to do gradient descent.
Sigmoid Unit
• The classic "soft threshold" that is used in
neural nets is referred to as a "sigmoid"
(meaning S-like) and is shown here.
– The variable z is the "total input" or
"activation" of a neuron, that is, the
weighted sum of all of its inputs.
– Note that when the input (z) is 0, the
sigmoid's value is 1/2.
– The sigmoid is applied to the weighted
inputs (including the threshold value
as before).
• There are actually many different types of
sigmoids that can be (and are) used in
neural networks.
– The sigmoid shown here is actually
called the logistic function.
Training
• The key property of the sigmoid is that it is differentiable.
– This means that we can use gradient based methods of
minimization for training.
• The output of a multi-layer net of sigmoid units is a function of
two vectors, the inputs (x) and the weights (w).
• The output of this function (y) varies smoothly with changes in
the input and, importantly, with changes in the weights.
Training
Training
• Given a set of training points, each of which specifies the net
inputs and the desired outputs, we can write an expression for the
training error, usually defined as the sum of the squared
differences between the actual output (given the weights) and the
desired output.
• The goal of training is to find a weight vector that minimizes the
training error.
• We could also use the mean squared error (MSE), which simply
divides the sum of the squared errors by the number of training
points instead of just 2. Since the number of training points is a
constant, the value for which we get the minimum is not affected.
Training
Gradient Descent
Online version:
We consider each time only
the error for one data item
1
E  ( y (x m , w )  y m ) 2
2
 w E  ( y (x m , w )  y m ) w y (x m , w )
where
 y
y 
 w y (x , w )  
,...,


w

w
n
 1
m
We changew as follows
w  w   w E
We've seen that the simplest
method for minimizing a
differentiable function is
gradient descent
(or ascent if we're maximizing).
Recall that we are trying to find
the weights that lead to a
minimum value of training error.
Here we see the gradient of the
training error as a function of the
weights.
The descent rule is basically to
change the weights by taking a
small step (determined by the
learning rate ) in the direction
opposite this gradient.
Gradient Descent – Single Unit
 y
y 
 w y (x , w )  
,...,


w

w
n
 1
i
n
z   wi xi
i
y y z

wi z wi
s ( z ) z

z wi
s ( z )

xi
z
1
y  s( z ) 
1  ez
Substituting in the equation
of previous slide we get (for
the arbitrary ith element):
s ( z )
wi  wi   ( y  y )
xi
z
m
E
m s ( z )

 (y  y )
z
z
wi   xi
Delta
rule
Derivative of the sigmoid
1
s( z ) 
1  ez
ds( z ) d

(1  e  z ) 1
dz
dz
 [(1  e  z )  2 ][e  z ]


z


1
e



z 
z 
1

e
1

e



 s ( z )(1  s ( z ))
Generalized Delta Rule
w14  4 y1
w24  4 y2
Now, let’s compute 4.
E
4 
z 4
E z5 E z6


z5 z 4 z6 z 4
z5
z6
 5
 6
z 4
z 4
 ...
z4 will influence E, only indirectly
through z5 and z6.
z5
z6
E E z5 E z6
4 


 5
 6
z 4 z5 z 4 z6 z 4
z 4
z 4
z5 y4
z6 y4
 5
 6
y4 z 4
y4 z 4
 (... w45 y4  ...) y4
 (... w46 y4  ...) y4
 5
 6
y4
z 4
y4
z 4
y4
y4
  5 w45
  6 w46
z 4
z 4
  5 w45
d ( z4 )
d ( z4 )
  6 w46
dz4
dz4
d ( z4 )

( 5 w45   6 w46 )
dz4
 s ( z 4 )(1  s ( z 4 ))( 5 w45   6 w46 )
 y4 (1  y4 )( 5 w45   6 w46 )
Generalized Delta Rule
 4  y4 (1  y4 )( 5 w45   6 w46 )
w14  4 y1
w24  4 y2
In general, for a hidden unit j we have
 j  y j (1  y j )

 k w j k
kdownstream ( j )
wi  j  wi  j   j yi
Generalized Delta Rule
For an output unit we have
E
n 
z n
win  n yi
E yn

yn z n
ds( z n ) E

dzn yn
E
 yn (1  yn )
yn
1

 ( yn  y m ) 2 
2


 yn (1  yn )
yn
 yn (1  yn )( yn  y m )
Backpropagation Algorithm
1.
2.
3.
4.
5.
6.
Initialize weights to small random values
Choose a random sample training item, say (xm, ym)
Compute total input zj and output yj for each unit (forward prop)
Compute n for output layer n = yn(1-yn)(yn-ynm)
Compute j for all preceding layers by backprop rule
Compute weight change by descent rule (repeat for all weights)
•
Note that each expression involves data local to a particular unit,
we don't have to look around summing things over the whole
network.
It is for this reason, simplicity, locality and, therefore, efficiency
that backpropagation has become the dominant paradigm for
training neural nets.
•
Training Neural Nets
• Now that we have looked at the basic mathematical techniques
for minimizing the training error of a neural net, we should step
back and look at the whole approach to training a neural net,
keeping in mind the potential problem of overfitting.
• Here we look at a methodology that attempts to minimize that
danger.
Training Neural Nets
Given: Data set, desired outputs and a neural net with m weights.
Find a setting for the weights that will give good predictive
performance on new data.
1. Split data set into three subsets:
i. Training set – used for adjusting weights
ii. Validation set – used to stop training
iii. Test set – used to evaluate performance
2. Pick random, small weights as initial values
3. Perform iterative minimization of error over training set
(backprop)
4. Stop when error on validation set reaches a minimum (to avoid
overfitting)
5. Repeat training (from step 2) several times (to avoid local
minima)
6. Use best weights to compute error on test set.
Backpropagation Example
w03
First do forward propagation:
Compute zi’s and yi’s.
y3
3
z3
-1
w13
1
w01
-1
 3  y3 (1  y3 )( y3  y m )
 2  y2 (1  y2 ) 3 w23
1  y1 (1  y1 ) 3 w13
w23
y1
y2
z1
z2
w21
w12
w11
2
w02
w22
x1
x2
-1
w03  w03  r 3 (1)
w02  w02  r 2 (1)
w01  w01  r1 (1)
w13  w13  3 y1
w12  w12  2 x1
w11  w11  1 x1
w23  w23  3 y2
w22  w22  2 x2
w21  w21 1 x2