Transcript Training
2806 Neural Computation Multilayer neural networks Lecture 4 2005 Ari Visa
Agenda Some historical notes Multilayer neural networks Backpropagation Error surfaces and feature mapping Speed ups Conclusions
Some historical notes Rosenblatt’s perceptron 1958: a single neuron with adjustable synaptic weights and bias Perceptron convergence theorem 1962 (
Rosenblatt
): If patterns used to train the perceptron are drawn from two linearly separable classes, then the algorithm converges and positions the decision surface in the form of hyperplane between two classes.
The limitations of networks implementing linear discriminants were well known in the 1950s and 1960s.
Some historical notes A single neuron -> the class of solutions that can be obtained is not general enough -> multilayer perceptron
Widrow & Hoff
(1960): least-mean square algorithm (LMS) = delta rule: three-layer networks designed by hand (fixed input-to-hidden weights, trained hidden-to-output weights).
The development of backpropagation: Kalman filtering (
Kalman
1960,
Bryson,Denham,Dreyfus
1963,1969).
Some historical notes
Werbos
: Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Science, Ph.D. thesis, Harvard University, 1974
Parker
1982,1985
Rumelhart, Hinton, Williams
: Learning internal representations by backpropagating errors, Nature 323(99),pp533-536,1986
Multilayer Neural Networks Input layer Hidden layer Output layer Bias unit = neuron net j =
w t j x
y j = f(net j ) f(net) = Sgn(net) z k = f(net k )
Multilayer Neural Networks
The XOR problem:
(x 1 OR x 2 ) AND NOT (x 1 AND x 2 ) y 1 : x 1 + x 2 +0.5 = 0 > 0 -> y 1 = 1, otherwise –1 y 2 : x 1 + x 2 -1.5 = 0 > 0 -> y 2 = 1, otherwise –1
Multilayer Neural Networks Any desired continuous function can be implemented by a three layer network given sufficient number of hidden units, proper nonlinearitiers and weights (Kolmogorov) Feedforward Learning (Demo chapter 11)
Multilayer Neural Networks Nonlinear multilayer networks How to set the weights of a three-layer neural network based on training patterns and the desired output?
Backpropagation Algorithm The LMS algorithm exists for linear systems Training error Learning rule
Backpropagation Algorithm Learning rule Hidden-to-output Input-to-hidden Note, that w ij are initialized with random values Demo Chapter 11
Backpropagation Algorithm Compare with LMS algoritms 1) Method of Steepest Descent The direction of steepest descent is in direction opposite to the gradient vector
g
= E(
w
)
w
(n+1) =
w
(n) –
g
(n) is the stepsize or learning-rate parameter
Backpropagation Algorithm
Training set
= a set of patterns with known labels
Stochastic training
from the training set = patterns are chosen randomly
Batch training
= all patterns are presented to the network before learning takes place
On-line protocol
= each pattern is presented once and only once, no memory in use
Epoch
= a single presentation of all patterns in the training set. The number of epochs is an indication of the relative amount of learning.
Backpropagation Algorithm Stopping criterion
Backpropagation Algorithm Learning set Validation set Test set Stopping criterion Learning curve, the average error per pattern Cross-validation
Error Surfaces and Feature Mapping Note, error backpropagation is based on gradient descent in a criterion function J(
w
) that is represented represented
Error Surfaces and Feature Mapping The total training error is minimized.
It usually decreases monotonically, even though this is not the case for the error on each individual pattern.
Error Surfaces and Feature Mapping Hidden-to-output weights ~ a linear discriminant Input-to-hidden weights ~ ”matched filter”
Practical Techniques for Improving Backpropagation How to improve convergence, performance, and results?
Neuron: Sigmoid function = centered zero and antisymmetric
Practical Techniques for Improving Backpropagation Scaling input variables = the input patterns should be shifted so that the average over the training set of each feature is zero.
= the full data set should be scaled to have the same variance in each feature component Note, this standardization can only be done for stochastic and batch learning!
Practical Techniques for Improving Backpropagation
When the training set is small
surrogate training patterns.
one can generate In
the absence of problem-specific information
, the
surrogate patterns
should be made by adding d-dimensional Gaussian noise to true training points, the category label should be left unchanged.
If we know about the source of variation among patterns we can
manufacture training data
.
The
number of hidden units
should be less than the total number of training points n, say roughly n/10.
Practical Techniques for Improving Backpropagation We cannot initialize the weights to 0.
Initializing weights = uniform learning -> choose weights randomly from a single distribution Input-to-hidden weights: -1 / d < w ij ,where d is the number of input units < + 1 / d Hidden-to-output weights: -1 / n H < w kj n H ,where d is the number of hidden units < + 1 /
Practical Techniques for Improving Backpropagation Learning Rates Demo Chapter 9 The optimal rate
Practical Techniques for Improving Backpropagation
Momentum
: allows the network to learn more quickly when plateaus in the error surface exist. Demo chapter 12. 0.9
Practical Techniques for Improving Backpropagation
Weight Decay
to avoid overfitting = to start with a network with too many weights and decay all the weights during training w new = w old (1 ), where 0 < < 1 The weights that are not needed for reducing the error function become smaller and smaller -> eliminated
Practical Techniques for Improving Backpropagation If we have insufficient training data for the desired classification accuracy.
Learning with hints
add output units for is to addressing an ancillary problem, one different from but related to the specific classification problem at hand.
Practical Techniques for Improving Backpropagation Stopped Training Number of hidden layers: typically 2-3.
Criterion Function: cross entropy, Minkowski Error
Practical Techniques for Improving Backpropagation Second-Order Methods to speed up the learning rate: Newton’s Method, demo chapter 9 Quickprop Conjugate Gradient Descent requires batch training, demo chapter 9
Practical Techniques for Improving Backpropagation 2) Newton’s method The idea is to minimize the quadratic approximation of the cost function E(
w
) around the current point
w
(n). Using a second-order Taylor series expansion of the cost function around the point
w
(n).
E
w
(n)
g
T (n)
w
(n) +½
w
T (n)
H
(n)
w
(n) g(n) is the m-by-1 gradient vector of the cost function E(
w
) evaluated at the point
w
(n). The matrix
H
(n) is the m-by-m Hessian matrix of E(
w
) (second derivative),
H
= ²E(
w
)
Practical Techniques for Improving Backpropagation
H
= ²E(
w
) requires the cost function E(
w
) to be twice continuously differentiable with respect to the elements of
w
. Differentiating E
w
(n)
g
T (n)
w
(n) +½
w
T (n)
H
(n)
w
(n) with respect to
w
, the change E(n) is minimized when
g
(n) +
H
(n)
w
(n) =
0 ->
w
(n) =
H
-1 (n)
g
(n)
w
(n+1) =
w
(n) +
w
(n)
w
(n+1) =
w
(n)+
H
-1 (n)
g
(n) where
H
-1 (n) is the inverse of the Hessian of E(
w
).
Practical Techniques for Improving Backpropagation Newton’s method converges quickly asymtotically and does not exhibit the zigzagging behavior.
Newton’s method requires that the Hessian H(n) has to be a positive definite matrix for all n!
Practical Techniques for Improving Backpropagation 3) Gauss-Newton Method It is applicable to a cost function that is expressed as the sum of error squares.
E(
w
) = ½ i=1 n e ² (i), note that all the error terms are calculated on the basis of a weight vector
w
fixed over the entire observation interval 1 that is i n.
The error signal e(i) is a function of the adjustable weight vector
w
. Given an operating point
w
(n), we linearize the dependence of e(i) on
w
writing e’(i,
w
) = e(i) + [ e(i)/
w
] T
w
=
w
(n) (
w
-
w
by (n)), i=1,2,...,n
Practical Techniques for Improving Backpropagation
e
’(n,
w
) =
e
(n) +
J
(n)(
w
-
w
(n)) where
e
(n) is the error vector
e
(n) = [e(1),e(2),...,e(n)] T and
J
(n) is the n-by-m Jacobian matrix of
e
(n) (The Jacobian
J
(n) is the transpose of the m-by-n gradient matrix
e
(n), where
e
(n) =[
e
(1),
e
(2), ...,
e
(n)].
w
(n+1) = arg min w {½
e
’(n,
w
) ²} ½
e
(n) ² +
e
T (n)
J
(n)(
w-w
(n)) + ½(
w w
(n)) T
J
T (n)
J
(n)(
w
-
w
(n)) = Differentiating the expression with respect
w
setting the result equal to zero and
Practical Techniques for Improving Backpropagation
J
T (n)
e
(n) +
J
T (n)
e
(n) (
w
-
w
(n)) =
0 w
(n+1) =
w
(n) – (
J
T (n)
J
(n)) -1
J
T (n)
e
(n) The Gauss-Newton requires only the Jacobian matrix of the error function
e
(n).
For the Gauss-Newton iteration to be computable, the matrix product
J
T (n)
J
(n) must be nonsigular.
J
T (n)
J
(n) is always nonnegative definite but to ensure that it is nonsingular, the Jacobian
J
(n) must have row
rank n
. -> add the diagonal matrix
I
to the matrix
J
T (n)
J
(n), the parameter is a small positive constant.
Practical Techniques for Improving Backpropagation
J
T (n)
J
(n)+
I
; positve definite for all n.
-> The Gauss-Newton method is
w
implemented in the following form: (n+1) =
w
(n) – (
J
T (n)
J
(n) +
I
) -1
J
T (n)
e
(n) This is the solution to the modified cost function: E(
w
) = ½{
w
-
w
(0) ²+ i=1 n e ² (i)} where
w
(0) is the initial value of
w.
Regularization, Complexity Adjustment and Pruning If we have too many weight and train too long > overfitting Wald Statistic: we can estimate the importance of a parameter in a model The Optimal Brain Damage, The Optimal Brain Surgeon: eliminate the parameter having the least importance
Summary Multilayer nonlinear neural networks trained by gradient descent methods such as backpropagation perform a maximum-likelihood estimation of the weight values in the model defined by the network topology.
f(net) at the hidden units allows the networks to form an arbitary decision boundary, so long as there are sufficiently many hidden units.