Slides from: Doug Gray, David Poole
What is a Neural Network?
• Information processing paradigm that is
inspired by the way biological nervous
systems, such as the brain, process
• A method of computing, based on the
interaction of multiple connected
What can a Neural Net do?
Compute a known function
Approximate an unknown function
Learn to do any of the above
A Neural Network generally
maps a set of inputs to a set
Number of inputs/outputs is
The Network itself is
composed of an arbitrary
number of nodes with an
Definition of a node:
• A node is an element
which performs the
y = fH(∑(wixi) + Wb)
Inputs are flexible
any real values
Highly correlated or independent
Target function may be discrete-valued, realvalued, or vectors of discrete or real values
Outputs are real numbers between 0 and 1
Resistant to errors in the training data
Long training time
The function produced can be difficult for
humans to interpret
Basic unit in a neural network
N inputs, x1 ... xn
Weights for each input, w1 ... wn
A bias input x0 (constant) and associated
Weighted sum of inputs, y = w0x0 + w1x1 + ...
A threshold function (activation function), i.e 1 if
y > 0, -1 if y <= 0
y = Σ wixi
1 if y >0
Typical Activation Functions
F(x) = 1 / (1 + e –x)
Using a nonlinear
approximates a linear
threshold allows a
network to approximate
Binary logic application
fH(x) = u(x) [linear threshold]
Wi = random(-1,1)
Y = u(W0X0 + W1X1 + Wb)
Now how do we train it?
Perception learning rule
ΔWi = η * (D – Y) * Xi
η = Learning Rate
D = Desired Output
Adjust weights based on how well the
current weights match an objective
Expose the network to the logical
Update the weights after each
As the output approaches the
desired output for all cases, ΔWi will
W0 W1 Wb
Network converges on a hyper-plane decision
X1 = (W0/W1)X0 + (Wb/W1)
Feed-forward neural networks
Feed-forward neural networks are the
most common models.
These are directed acyclic graphs:
Neural Network for the news
Axiomatizing the Network
The values of the attributes are real numbers.
Thirteen parameters w0; … ;w12 are real numbers.
The attributes h1 and h2 correspond to the values of
There are 13 real numbers to be learned. The
hypothesis space is thus a 13-dimensional real space.
Each point in this 13-dimensional space corresponds
to a particular logic program that predicts a value for
reads given known, new, short, and home
Neural Network Learning
Aim of neural network learning: given a set
of examples, find parameter settings that
minimize the error.
Back-propagation learning is gradient
descent search through the parameter
space to minimize the sum-of-squares
A network, including all units and their
Learning Rate (constant of proportionality of
gradient descent search)
Initial values for the parameters
A set of classified training data
Output: Updated values for the parameters
evaluate the network on each example given
the current parameter settings
determine the derivative of the error for each
change each parameter in proportion to its
until the stopping criteria is met
Gradient Descent for Neural Net
Bias in neural networks and
It’s easy for a neural network to represent “at least
two of I1, …, Ik are true”:
w0 w1 wk
This concept forms a large decision tree.
Consider representing a conditional: “If c then a
Simple in a decision tree.
Needs a complicated neural network to represent
(c ^ a) V (~c ^ b).
Neural Networks and Logic
Meaning is attached to the input and
There is no a priori meaning associated
with the hidden units.
What the hidden units actually represent is
something that’s learned.