Transcript Artificial Neural Networks

Artificial Neural
Networks
KONG DA, XUEYU LEI & PAUL MCKAY
Digit Recognition
Convolutional Neural
Network
Inspired by the visual
cortex
Our example:
Handwritten digit
recognition
Reference: LeCun et al. Back propagation Applied to Handwritten Zip Code Recognition. 1989
Method
Back Propagation:
1. Propagation:
2.Weight update:
http://en.wikipedia.org/wiki/Backpropagation
http://tex.stackexchange.com/questions/16232
6/drawing-back-propagation-neural-network
Back propagation
Le Cun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., and Jackel, L. D.
(1990). Back-Propagation Applied to Handwritten Zipcode Recognition. Neural Computation, 1(4).
Overfitting
Bias due to experience
https://www.youtube.com/watch?v=ZgqsaDnsEq8
Overfitting
http://pingax.com/regularization-implementation-r/
Possible solutions
Le Cun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., and Jackel, L. D.
(1990). Back-Propagation Applied to Handwritten Zipcode Recognition. Neural Computation, 1(4).
Demonstration
A neural network based on
back-propagation achieves an high
accuracy on a modified NIST database
of hand-written digits
Demonstration
Demonstration
•Learning Rate(LR)
•Mean Square Error (MSE)
•Accuracy
Learning Rate
Learning rate=0.0001
Learning rate=0.001
Learning rate=0.01
MSE
The left-hand
axis is for MSE
the right-hand
axis others
Back-propagation progress in one
epoch
•nitial learning rate (eta) = 0.001
•Minimum learning rate (eta) =
0.00005
•Rate of decay for learning rate
(eta) = 0.794183335
•Decay rate is applied after this
number of backprops = 120000
http://www.codeproject.com/Articles/16650/Neural-Network-for-Recognition-of-Handwritten-Digi#Backpropagation
Accuracy
Total number of testing set:10000
Total number of errors: 74 (non-distorted)
For each pattern:
No
Expected value => misrecognized value
Accuracy=99.26%
http://www.codeproject.com/Articles/16650/Neural-Network-for-Recognition-of-Handwritten-Digi#Backpropagation
Multi-digit number recognition
Large-scale deep neural network
11 convolutional layers
Street View House Numbers
Per-digit recognition: 97.84%
Tens of millions of street number
annotations
Multi-digit recognition: >90%
Reference: Goodfellow et al. Multi-digit Number Recognition from Street View Imagery using Deep
Convolutional Neural Networks. 2013
CAPTCHA puzzles
Large-scale deep neural network
9 convolutional layers
Hardest category: 99.8%
Reference: Goodfellow et al. Multi-digit Number Recognition from Street View Imagery using Deep
Convolutional Neural Networks (2013)