Transcript Learning Feature Hierarchies and Deep Learning

Advanced topics
Andrew Ng
Outline
• Self-taught learning
• Learning feature hierarchies
(Deep learning)
• Scaling up
Andrew Ng
Self-taught learning
Andrew Ng
Supervised learning
Cars
Motorcycles
Testing:
What is this?
Andrew Ng
Semi-supervised learning
Unlabeled images (all cars/motorcycles)
Testing:
What is this?
Car
Motorcycle
Andrew Ng
Self-taught learning
Unlabeled images (random internet images)
Testing:
What is this?
Car
Motorcycle
Andrew Ng
Self-taught learning
Sparse
coding,
LCC, etc.
f1, f2, …, fk
If have labeled
training set is small,
Use learned fcan
…, fk to
represent training/test sets.
1, f2, give
huge
performance boost.
Using f1, f2, …, fk
Car
a1, a2, …, ak
Motorcycle
Andrew Ng
Learning feature
hierarchies/Deep learning
Andrew Ng
Why feature hierarchies
object models
object parts
(combination
of edges)
edges
pixels
Andrew Ng
Deep learning algorithms
• Stack sparse coding algorithm
• Deep Belief Network (DBN) (Hinton)
• Deep sparse autoencoders (Bengio)
[Other related work: LeCun, Lee, Yuille, Ng
…]
Andrew Ng
Deep learning with autoencoders
• Logistic regression
• Neural network
• Sparse autoencoder
• Deep autoencoder
Andrew Ng
Logistic regression
Logistic regression has a learned parameter vector q.
On input x, it outputs:
where
x1
Draw a logistic
regression unit
as:
x2
x3
+1
Andrew Ng
Neural Network
String a lot of logistic units together. Example 3 layer
network:
x1
a1
a2
x2
a3
x3
Layer 3
+1
+1
Layer 1
Layer 3
Andrew Ng
Neural Network
Example 4 layer network with 2 output units:
x1
x2
x3
+1
+1
+1
Layer 1
Layer 2
Layer 4
Layer 3
Andrew Ng
Neural Network example
[Courtesy of Yann LeCun]
Andrew Ng
Training a neural network
Given training set (x1, y1), (x2, y2), (x3, y3 ), ….
Adjust parameters q (for every node) to make:
(Use gradient descent. “Backpropagation” algorithm.
Susceptible to local optima.)
Andrew Ng
Unsupervised feature learning with a neural network
x1
x1
x2
x2
x3
x4
x5
a1
a2
a3
x3
+1
x5
Layer 1
x6
Layer 2
Network is trained to
output the input (learn
identify function).
x4
+1
x6
Autoencoder.
Layer 3
Trivial solution unless:
- Constrain number of
units in Layer 2 (learn
compressed
representation), or
- Constrain Layer 2 to
be sparse.
Andrew Ng
Unsupervised feature learning with a neural network
Training a sparse autoencoder.
a1
Given unlabeled training set x1, x2, …
Reconstruction
error term
a2
a3
L1 sparsity term
Andrew Ng
Unsupervised feature learning with a neural network
x1
x1
x2
x2
a1
x3
x4
x5
x3
a2
a3
x4
x5
+1
x6
+1
x6
Layer 2
Layer 3
Layer 1
Andrew Ng
Unsupervised feature learning with a neural network
x1
x2
a1
x3
x4
x5
a2
a3
+1
x6
+1
New representation for input.
Layer 2
Layer 1
Andrew Ng
Unsupervised feature learning with a neural network
x1
x2
a1
x3
x4
x5
a2
a3
+1
x6
+1
Layer 2
Layer 1
Andrew Ng
Unsupervised feature learning with a neural network
x1
x2
a1
b1
a2
b2
a3
b3
+1
+1
x3
x4
x5
x6
+1
Train parameters so that
subject to bi’s being sparse.
,
Andrew Ng
Unsupervised feature learning with a neural network
x1
x2
a1
b1
a2
b2
a3
b3
+1
+1
x3
x4
x5
x6
+1
Train parameters so that
subject to bi’s being sparse.
,
Andrew Ng
Unsupervised feature learning with a neural network
x1
x2
a1
b1
a2
b2
a3
b3
+1
+1
x3
x4
x5
x6
+1
Train parameters so that
subject to bi’s being sparse.
,
Andrew Ng
Unsupervised feature learning with a neural network
x1
x2
a1
b1
a2
b2
a3
b3
+1
+1
x3
x4
x5
x6
New representation for input.
+1
Andrew Ng
Unsupervised feature learning with a neural network
x1
x2
a1
b1
a2
b2
a3
b3
+1
+1
x3
x4
x5
x6
+1
Andrew Ng
Unsupervised feature learning with a neural network
x1
x2
a1
b1
c1
a2
b2
c2
a3
b3
c3
+1
+1
+1
x3
x4
x5
x6
+1
Andrew Ng
Unsupervised feature learning with a neural network
x1
x2
a1
b1
c1
a2
b2
c2
a3
b3
c3
+1
+1
+1
x3
x4
x5
x6
New representation
for input.
+1
Use [c1, c3, c3] as representation to feed to learning algorithm.
Andrew Ng
Deep Belief Net
Deep Belief Net (DBN) is another algorithm
for learning a feature hierarchy.
Building block: 2-layer graphical model
(Restricted Boltzmann Machine).
Can then learn additional layers one at a
time.
Andrew Ng
Restricted Boltzmann machine (RBM)
a1
x1
a2
x2
a3
x3
Layer 2. [a1, a2, a3]
(binary-valued)
x4
Input [x1, x2, x3, x4]
MRF with joint distribution:
Use Gibbs sampling for inference.
Given observed inputs x, want maximum likelihood estimation:
Andrew Ng
Restricted Boltzmann machine (RBM)
a1
x1
a2
x2
a3
x3
Layer 2. [a1, a2, a3]
(binary-valued)
x4
Input [x1, x2, x3, x4]
Gradient ascent on log P(x) :
[xiaj]obs from fixing x to observed value, and sampling a from P(a|x).
[xiaj]prior from running Gibbs sampling to convergence.
Adding sparsity constraint on ai’s usually improves results.
Andrew Ng
Deep Belief Network
Similar to a sparse autoencoder in many ways.
Stack RBMs on top of each other to get DBN.
Layer 3. [b1, b2, b3]
Layer 2. [a1, a2, a3]
Input [x1, x2, x3, x4]
Andrew Ng
Deep Belief Network
Layer 4. [c1, c2, c3]
Layer 3. [b1, b2, b3]
Layer 2. [a1, a2, a3]
Input [x1, x2, x3, x4]
Andrew Ng
Deep learning examples
Andrew Ng
Convolutional DBN for audio
Max pooling unit
Detection units
Spectrogram
Andrew Ng
Convolutional DBN for audio
Spectrogram
Andrew Ng
Probabilistic max pooling
Convolutional DBN:
Convolutional Neural net:
max {x1, x2, x3, x4}
0
max {x1, x2, x3, x4}
0
0
0
X1 X2
0
X3 X4
Where xi are {0,1}, and mutually
exclusive. Thus, 5 possible cases:
0
0
1
1
0
0
0
1 0
0
0
0
1
1
X1
X2
X3
0
0
1
X4
Where xi are real numbers.
0
0
1 0
0
0
0
1
Collapse 2n configurations into n+1
configurations. Permits bottom up and
top down inference.
Andrew Ng
Convolutional DBN for audio
Spectrogram
Andrew Ng
Convolutional DBN for audio
Max pooling
Detection units
Max pooling
Detection units
Second CDBN
layer
One CDBN
layer
Andrew Ng
CDBNs for speech
Learned first-layer bases
Andrew Ng
Convolutional DBN for Images
‘’max-pooling’’ node (binary)
Max-pooling layer P
Detection layer H
Hidden nodes (binary)
Wk
“Filter” weights (shared)
Input data V
Andrew Ng
Convolutional DBN on face images
object models
object parts
(combination
of edges)
edges
pixels
Andrew Ng
Learning of object parts
Examples of learned object parts from object categories
Faces
Cars
Elephants
Chairs
Andrew Ng
Training on multiple objects
Trained on 4 classes (cars, faces, motorbikes, airplanes).
Second layer: Shared-features and object-specific features.
Third layer: More specific features.
Plot of H(class|neuron active)
Andrew Ng
Hierarchical probabilistic inference
Generating posterior samples from faces by “filling in” experiments
(cf. Lee and Mumford, 2003). Combine bottom-up and top-down inference.
Input images
Samples from
feedforward
Inference
(control)
Samples from
Full posterior
inference
Andrew Ng
Key issue in feature
learning: Scaling up
Andrew Ng
Scaling up with graphics processors
US$ 250
NVIDIA GPU
Peak
GFlops
Intel CPU
2003
2004
2005
2006
2007
2008
(Source: NVIDIA CUDA Programming Guide)
Andrew Ng
Scaling up with GPUs
Approx. number of parameters (millions):
Using GPU (Raina et al., 2009)
Andrew Ng
Unsupervised feature
learning: Does it work?
Andrew Ng
Audio
State-of-the-art task performance
TIMIT Phone classification
Accuracy
TIMIT Speaker identification
Accuracy
Prior art (Clarkson et al.,1999)
79.6%
Prior art (Reynolds, 1995)
99.7%
Stanford Feature learning
80.3%
Stanford Feature learning
100.0%
Images
CIFAR Object classification Accuracy
NORB Object classification Accuracy
Prior art (Yu and Zhang, 2010)
74.5%
Prior art (Ranzato et al., 2009)
94.4%
Stanford Feature learning
75.5%
Stanford Feature learning
96.2%
UCF activity classification
Accuracy
Hollywood2 classification
Accuracy
Prior art (Kalser et al., 2008)
86%
Prior art (Laptev, 2004)
47%
Stanford Feature learning
87%
Stanford Feature learning
50%
Video
Multimodal (audio/video)
AVLetters Lip reading
Accuracy
Prior art (Zhao et al., 2009)
58.9%
Stanford Feature learning
63.1%
Andrew Ng
Summary
• Instead of hand-tuning features, use
unsupervised feature learning!
• Sparse coding, LCC.
• Advanced topics:
–Self-taught learning
–Deep learning
–Scaling up
Andrew Ng
Other resources
Workshop page:
http://ufldl.stanford.edu/eccv10-tutorial/
• Code for Sparse coding, LCC.
• References.
• Full online tutorial.
Andrew Ng