Transcript Intelligent Information Retrieval and Web Search

Machine Learning
Neural Networks
1
Neural Networks
• NN belong to the category of trained
classification models
• The learned classification model is an
algebraic function (or a set of functions),
rather than a boolean function as for DTrees
• Linear function for Perceptron algorithm,
non-linear for Backpropagation algorithm
• Both features and the output class are
allowed to be real valued
2
Neural Networks
• Analogy to biological neural systems, the most
robust learning systems we know.
• Attempt to understand natural biological systems
through computational modeling.
• Massive parallelism allows for computational
efficiency.
• Help understand “distributed” nature of neural
representations (rather than “localist”
representation) that allow robustness and graceful
degradation.
• Intelligent behavior as an “emergent” property
of large number of simple units rather than
from explicitly encoded symbolic rules and
algorithms.
3
Neural Speed Constraints
• Neurons have a “switching time” on the order of a
few milliseconds, compared to nanoseconds for
current computing hardware.
• However, neural systems can perform complex
cognitive tasks (vision, speech understanding) in
tenths of a second.
• Only time for performing 100 serial steps in this
time frame, compared to orders of magnitude
more for current computers.
• Must be exploiting “massive parallelism.”
• Human brain has about 1011 neurons with an
average of 104 connections each.
4
Neural Network Learning
• Learning approach based on modeling
adaptation in biological neural systems.
• Perceptron: Initial algorithm for learning
simple neural networks (single layer)
developed in the 1950’s.
• Backpropagation: More complex algorithm
for learning multi-layer neural networks
developed in the 1980’s.
5
Real Neurons
• Cell structures
–
–
–
–
Cell body
Dendrites
Axon
Synaptic terminals
6
Neural Communication
• Electrical potential across cell membrane exhibits spikes
called action potentials.
• Spike originates in cell body, travels down
axon, and causes synaptic terminals to
release neurotransmitters.
• Chemical diffuses across synapse to
dendrites of other neurons.
• Neurotransmitters can be excitatory or
inhibitory.
• If net input of neurotransmitters to a neuron from other
neurons is excitatory and exceeds some threshold, it fires
an action potential.
7
Neural connections
8
Real Neural Learning
• Synapses change size and strength with
experience.
• Hebbian learning: When two connected
neurons are firing at the same time, the
strength of the synapse between them
increases.
• “Neurons that fire together, wire together.”
9
Artificial Neuron Model
• Model network as a graph with cells as nodes, and synaptic
connections as weighted edges from node i to node j, wji
• Model net input to cell as
net j = å w ji xi
i
Tj
• Cell output is:
0 if net j  T j
oj 
1 if neti  T j
(activation function φ is a
threshold function Tj )
oj
1
0
Tj
netj
10
Perceptron schema
Perceptron training phase = estimating edge weights wji and threshold Tj
x1
x2
0 if net j  T j
oj 
1 if neti  T j
w j1
Threshold
Tj
∑
xn
w jn
oj
netj
Function to be learned is:
11
Perceptron as a Linear Separator
• Since perceptron uses linear threshold function, it is searching
for a linear separator that discriminates the classes.
w12
2
x2
3
w13
1
w31 x1 + w32 x2 > T3
??
w31
T3
x2 > x1 +
w32
w32
x1
Or hyperplane in
n-dimensional space
12
Concept Perceptron Cannot Learn
• Cannot learn exclusive-or, or parity
function in general (not linearly separable
functions!).
o3
1
+
??
–
–
+
0
1
o2
13
Perceptron Training
• Assume supervised training examples
giving the desired output, given a set of
known instances.
• Objective: Learn synaptic weights (wij) so
that each instance produces the correct
output for each example.
• Perceptron uses iterative update algorithm
to learn a correct set of weights.
14
Perceptron Learning Rule
• Initial weights selected at random.
• Update weights by: w(t ) = w(t-1)+ h(t - o )x
0 if net j  T j
ji
ji
j
j
oj 
i
1 if neti  T j
Dw ji = h(Errj )xi
where η is a constant named the “learning rate”
tj is the “teacher” specified output for unit j, (tj -oj) is the
error in iteration t
Equivalent to rules:
– If output is correct do nothing (Errj=0).
– If output is “high” (oj=1, tj=0, Errj=-1), lower weights on
active inputs
– If output is “low” (Errj=+1), increase weights on active
inputs
t
t-1
T
=
T
h
(t
o
)
j
j
j
j
15
• Also adjust threshold to compensate:
Perceptron Learning Algorithm
• Training set D of vectors and their correct
classification <d (x1..xn), tj>
• Iteratively update weights until convergence.
Initialize weights to random values
Until outputs of all training examples are correct
For each training pair, Ej <d (x1,x2,xn),tj>, do:
Compute current output oj for Ej given its inputs
Compare current output to target value, tj , for Ej
Update synaptic weights and threshold using learning rule
• Each execution of the outer loop is typically
called an epoch.
16
Perceptron Limits
• System obviously cannot learn concepts it
cannot represent.
• Minksy and Papert (1969) wrote a book
analyzing the perceptron and demonstrating
many functions it could not learn.
• These results discouraged further research
on neural nets; and symbolic AI became the
dominate paradigm.
17
Perceptron Convergence
and Cycling Theorems
• Perceptron convergence theorem: If the data is
linearly separable and therefore a set of weights
exist that are consistent with the data, then the
Perceptron algorithm will eventually converge to a
consistent set of weights.
• Perceptron cycling theorem: If the data is not
linearly separable, the Perceptron algorithm will
eventually repeat a set of weights and threshold at
the end of some epoch and therefore enter an
infinite loop.
18
Perceptron as Hill Climbing
• The hypothesis space being search is a set of weights and a
threshold.
• Objective is to minimize classification error on the training set.
• Perceptron effectively does hill-climbing (gradient descent) in
this space, changing the weights a small amount at each point
to decrease training set error. (see later on gradient descent)
• For a single model neuron, the space is well behaved with a
single minima.
training
error
0
weights
19
Example: Learn a NOR function
w1 x1 + w2 x2 > u
(u = Threshold)
w = w + 0.05(t j - o )o u = u - 0.05(t j - o )
(t )
i
(t-1)
i
(t-1)
i
j
t
j
t-1
j
(t-1)
j
20
21
22
23
24
25
After 20 iterations
26
Perceptron Performance
• In practice, converges fairly quickly for
linearly separable data.
• Can effectively use even incompletely
converged results when only a few outliers
are misclassified.
• Experimentally, Perceptron does quite well
on many benchmark data sets.
27
Multi-Layer Networks
• Multi-layer networks can represent arbitrary functions
• A typical multi-layer network consists of an input, hidden
and output layer, each fully connected to the next, with
activation feeding forward.
output
hidden
activation
input
• The weights determine the function computed.
28
Hill-Climbing in Multi-Layer Nets
• Since “greed is good” hill-climbing can be used to learn
multi-layer networks in practice, although its theoretical
limits are clear.
• However, to do gradient descent with multiple neurons, we
need the output of a unit to be a differentiable function of
its input and weights.
• Standard linear threshold function is not differentiable
at the threshold.
training
error
oi
1
0
Tj
netj
weights 29
Differentiable Output Function
• Need non-linear output function to move beyond linear
functions.
– Standard solution is to use the non-linear, differentiable
sigmoidal “logistic” function:
o j = s (net j ) =
1
1+ e
-(net j -T j )
net j = å w ji xi = å w ji oi
i
1
0
Tj
netj
i
Where oi are the input to node j and also the output of previous nodes
30
• Structure of a node:
threshold
Thresholding function limits node output
(Tj=0 in the example):
31
Feeding data through the net
(1  0.25) + (0.5  (-1.5)) = 0.25 + (-0.75) = - 0.5
thresholding:
1
 0.3775
0 .5
1 e
32
Gradient and gradient descent
• The gradient of a scalar field is a vector
field that points in the direction of the
greatest rate of increase of the scalar field,
and whose magnitude is that rate of
increase.
• In simple terms, the variation of any
quantity – e.g. an error function - can be
represented (e.g. graphically) by a slope.
The gradient represents the steepness and
direction of that slope.
33
Gradient and gradient descent
• To find a local minimum of a
function (e.g. f=error(x)) using
gradient descent, one takes steps
proportional to the negative of the
gradient (or of the approximate
gradient) of the function at the
current point.
• Matematically, this is obtained by
computing the derivative of the
squared (error) function over
weights W , which is for the
multilayer NN:
E(W )   (tkd  okd )2
dD kK
34
Gradient Descent
• The objective is to minimize squared error:
E(W ) = å å (t jd - o jd )2
dÎD jÎK
where D is the set of training examples, W are weights, K is
the set of output units, tjd and ojd are, respectively, the
teacher’s and current output for output node Oj for the
example: d : < (x1d , x2d ..., xid ,....xnd ),(t1d , t2d ,..t jd ..tkd ) >
• Learning rule to change weights to minimize error is:
E
w ji  
w ji
• Weights are updated proportionally to the derivative of the
error (move in the opposite direction of error gradient)
35
Computing the derivative
¶E dE do j ¶net j
=
¶w ji do j dnet j ¶w ji
E = (t j - o j )
o j = s (net j ) =
net j = å w ji xi
tj is a costant value
1
1+ e
-(net j -Tj )
i
How the error changes when weights change
How the error changes when output changes
How the output changes when weighted sum change
How the weighet sum changes when weights change
36
Computing the derivative
¶(net j )
w ji
¶(net j )
xi
=
¶(å w ji xi )
i
w ji
= xi
¶E dE do j ¶net j
=
¶w ji do j dnet j ¶w ji
= w ji
1
d(
)
-net j -Tj
d(o j )
= d( f (o j )) = 1+ e
= o j (1- o j )
d(net j )
net j
dE
d 1
=
(t - o j )2 = o j - t
do j do j 2
¶E
= o j (1- o j )(o j - t)xi
¶w ji
E(W ) = å å (t jd - o jd )2
dÎD jÎK
d(g(f(x))=d(g(x)d(f(x))
37
Backpropagation Learning Rule
Dw ji = hd j oi
• Each weight changed by:
i
oi
j
oj
wji
d j = o j (1- o j )(t j - o j )
if j is an output unit
 j  o j (1  o j ) k wkj
if j is a hidden unit
k
where η is a constant called the learning rate
tj is the correct “teacher” output for unit j
δj is the error measure for unit j
38
Error backpropagation in the hidden
nodes
 j  o j (1  o j ) k wkj
if j is a hidden unit
k
Notice that the error is (t-ok) for an output node,
while in a hidden node is the weighted sum
of the errors generated by all the output nodes
to which it is connected! (the error
BACKPROPAGATES)
39
Hidden nodes are “responsible” of errors on output nodes for
a fraction depending on the weight of their connections to
output nodes
δ=0.4
δ=0.2
δ=oH2(1-oH2)[0.1×0.4+1.17×0.2]
40
Backpropagation Training Algorithm
Create the 3-layer network with H hidden units with full connectivity
between layers. Set weights to small random real values.
Until all training examples produce the correct value (within ε), or
mean squared error ceases to decrease, or other termination criteria:
Begin epoch
For each training example, d, do:
Calculate network output for d’s input values
Compute error between current output and correct output for d
Update weights by backpropagating error and using learning rule
End epoch
41
Example
Oj one of the K output nodes
hk the hidden nodes
nj the input nodes
o j = s (x j1w j1 + x j 2w j 2) = s (oh1w j1 + oh w j2)
2
oh = s (xn wn h + xn wn h +.... + xn wn h )
oj
1
n1
2
wj1
wj2
h1
h2
…...
nj
d1: x11 x12…x1n
1 1
2
N
2 1
N 1
oh = s (xn wn h + xn wn h +.... + xn wn h )
Oj
…….
1
1
1 2
2
N
2 2
N 2
xn = on = s (x11 )....... xn = on = s (x1N )
1
…..
nn
1
N
N
Step 1: feed with first example
and compute all oj with initial
weights
inputsuccessor node = outputantecedent node
Example (cont’d)
d j = o j (1- o j )(t1 - o j )
oj
O
h1hidden
n1in
oh1 (1- oh1 )w j1d j
wj2
wj1
…….
dh1 = x j1 (1- x j1 )w j1d j =
out
j
dh2 = oh2(1- oh2 )w j 2d j
h2hidden
…...
…..
nnin
Compute the error on the output node, and backpropagate
computation on hidden nodes
Example (cont’d)
oj
O out
j
h1hidden
wj1
wh1n1
…….
wj2
…and on input nodes
h2hidden
wh2n1
…...
…..
n1in
nnin
d (n1in ) = o1in (1- o1in )(wh1n1dh1 + wh2n1dh2 )
......
d (nNin ) = oNin (1- oNin )(wh1n dh1 + wh2n dh2 )
N
N
Example (cont’d): update weights and
iterate
w j1 ¬ w j1 + hd j x j 1
w j2 ¬ w j2 + hd j x j 2
wh1n1 ¬ wh1n1 + hd h1x h1 ......
n1
•Update all weights
•Consider the second example
•Compute input and output in all nodes
•Compute errors on outpus and on all nodes
•Re-update weights
•Until all examples have been considered (Epoch)
•Repeat for n Epochs, until convergence
Another example (weight updates)
• Training set
–
–
–
–
((0.1, 0.1), 0.1)
((0.1, 0.9), 0.9)
((0.9, 0.1), 0.9)
((0.9, 0.9), 0.1)
0,1
-0,1
-0,1
3
-0,1
0,1
Initial weights
1
2
0,1
1. Compute output and error on output
node
First example:
((0.1, 0.1), 0.1)
0,1
0,1
s (1) = s (w1x1+ w2x2) =
-0,1
1
-0,1
0,1
-0,1
1
1+ e-(w1x1+w2x2)
3
0,5
0,1
0,1
2
0,5
0,1
0,5
1
Err = (0,1- 0,5) 2 = 0,08
2
d j = o j (1- o j )(t j - o j )
d (3) = 0,5(1- 0,5)(0.1- 0,5) = -0,5´ 0,5´ 0,40 = -0,1
2. Backpropagate error
d h = oh (1- oh ) å w khd k
n k ÎO
d (1) = 0,5(1- 0,5)[ 0,1(-0,1)] = -0,025
d (2) = 0,5(1- 0,5)[ 0,1(0,1)] = 0,025
1
-0,025
-0,1 0,1
0,1
0,5
2
0,1
0,025
0,5
3
0,5
3. Update weights
0,1
0,1
1
-0,025
-0,1
-0,1
0,5
-0,1
w 31 = -0,1+ 0,2 ´ 0,5 ´ (-0,1) = -0,11
w 32 = 0,1+ 0,2 ´ 0,5 ´ (-0,1) = 0,09
0,1
w1i1 = 0,1+ 0,2 ´ 0,1´ (0,025) = 0,1005
w1i2 = -0,1+ 0,2 ´ 0,1´ (0,025) = -0,0995
w 2i1 = -0,1+ 0,2 ´ 0,1´ (-0,025) = -0,1005
w 2i2 = 0,1+ 0,2 ´ 0,1´ (-0,025) = 0,0995
-0,1 0,1
3
0,1
0,1
2
0,025
0,5
0,5
Read second input from D
2. ((0,1,0,9), 0,9)
0,1
0,1005
1
-0,11
-0,0995
0,48
3
0,9
-0,1005
0,9
0,0995
0,09
2
0,501
0,52
..etc!!
Yet another example
• First calculate error of output units and use this to
change the top layer of weights.
Current output: oj=0.2
Correct output: tj=1.0
Error δj = oj(1–oj)(tj–oj)
0.2(1–0.2)(1–0.2)=0.128
output
Update weights into j
w ji   j oi
hidden
input
51
Error Backpropagation
• Next calculate error for hidden units based on
errors on the output units it feeds into.
output
 j  o j (1  o j ) k wkj
k
hidden
input
52
Error Backpropagation
• Finally update bottom layer of weights based on
errors calculated for hidden units.
output
 j  o j (1  o j ) k wkj
k
Update weights into j
hidden
w ji   j oi
input
53
Example: Voice Recognition
• Task: Learn to discriminate between two
different voices saying “Hello”
• Data
– Sources
• Steve Simpson
• David Raubenheimer
– Format
• Frequency distribution (60 bins)
• Network architecture
– Feed forward network
• 60 input (one for each frequency bin)
• 6 hidden
• 2 output (0-1 for “Steve”, 1-0 for “David”)
• Presenting the data
Steve
David
• Presenting the data (untrained network)
Steve
0.43
0.26
David
0.73
0.55
• Calculate error
Steve
0.43 – 0
= 0.43
0.26 –1
= 0.74
0.73 – 1
= 0.27
0.55 – 0
= 0.55
David
• Backprop error and adjust weights
Steve
0.43 – 0
= 0.43
0.26 – 1
= 0.74
1.17
David
0.73 – 1
= 0.27
0.55 – 0
= 0.55
0.82
• Repeat process (sweep) for all training pairs
–
–
–
–
Present data
Calculate error
Backpropagate error
Adjust weights
• Repeat process multiple times
• Presenting the data (trained network)
Steve
0.01
0.99
David
0.99
0.01
Comments on Training Algorithm
• Not guaranteed to converge to zero training error,
may converge to local optima or oscillate
indefinitely.
• However, in practice, does converge to low error
for many large networks on real data.
• Many epochs (thousands) may be required, hours
or days of training for large networks.
• To avoid local-minima problems, run several
trials starting with different random weights
(random restarts).
– Take results of trial with lowest training set error.
– Build a committee of results from multiple trials
(possibly weighting votes by training set accuracy).
62
Representational Power
• Boolean functions: Any boolean function can be
represented by a two-layer network with sufficient
hidden units.
• Continuous functions: Any bounded continuous
function can be approximated with arbitrarily
small error by a two-layer network.
– Sigmoid functions can act as a set of basis functions for
composing more complex functions, like sine waves in
Fourier analysis.
• Arbitrary function: Any function can be
approximated to arbitrary accuracy by a threelayer network.
63
Over-Training Prevention
error
• Running too many epochs can result in over-fitting.
on test data
on training data
0
# training epochs
• Keep a hold-out validation set and test accuracy on it after
every epoch. Stop training when additional epochs
actually increase validation error.
• To avoid losing training data for validation:
– Use internal 10-fold CV on the training set to compute the average
number of epochs that maximizes generalization accuracy.
– Train final network on complete training set for this many epochs.
64
Determining the Best
Number of Hidden Units
error
• Too few hidden units prevents the network from
adequately fitting the data.
• Too many hidden units can result in over-fitting.
on test data
on training data
0
# hidden units
• Use internal cross-validation to empirically determine an
optimal number of hidden units.
65
Successful Applications
• Text to Speech (NetTalk)
• Fraud detection
• Financial Applications
– HNC (eventually bought by Fair Isaac)
• Chemical Plant Control
– Pavillion Technologies
• Automated Vehicles
• Game Playing
– Neurogammon
• Handwriting recognition
66