Neural network architectures and learning algorithms

Author : Bogdan M. Wilamowski Source : IEEE INDUSTRIAL ELECTRONICS MAGAZINE Date : 2011/11/22 Presenter : 林哲緯 1

Outline

• • • • • Neural Architectures Parity-N Problem Suitable Architectures Use Minimum Network Size Conclusion 2

Neural Architectures

Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0) 3

Neural Architectures

Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0) 4

Neural Architectures

Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0) 5

error back propagation(EBP) algorithm • error back propagation(EBP) algorithm – multilayer perceptron (MLP) Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0) 6

multilayer perceptron (MLP)

MLP-type architecture 3-3-4-1(without connections across layers) Neural network architectures and learning algorithms, Wilamowski, B.M.

7

neuron by neuron(NBN) algorithm

• neuron by neuron(NBN) algorithm – bridged multilayer perceptron (BMLP) – fully connected cascade (FCC) arbitrarily connected network Neural network architectures and learning algorithms, Wilamowski, B.M.

8

neuron by neuron(NBN) algorithm

• Levenberg–Marquardt(LM) algorithm – Improve nonlinear function of least square – Forward & Backward Computation • Jacobian Matrix – Forward-Only Computation 9

bridged multilayer perceptron (BMLP) BMLP architecture 3=3=4=1(with connections across layers marked by dotted lines) Neural network architectures and learning algorithms, Wilamowski, B.M.

10

fully connected cascade (FCC) Bipolar neural network for parity-8 problem in a FCC architecture Neural network architectures and learning algorithms, Wilamowski, B.M.

11

Outline

• • • • • Neural Architectures Parity-N Problem Suitable Architectures Use Minimum Network Size Conclusion 12

parity-8 problem

MLP 8*9 + 9 = 81 weights BMLP 4*9 + 8 + 4 + 1 = 49 weights Neural network architectures and learning algorithms, Wilamowski, B.M.

13

parity-8 problem

9 + 10 + 11 + 12 = 42 weights Neural network architectures and learning algorithms, Wilamowski, B.M.

14

parity-17 problem

• • • MLP architecture needs 18 neurons BMLP architecture with connections across hidden layers needs 9 neurons FCC architecture needs only 5 neurons 15

parity-N problem

• MLP architectures nn = neurons nw = weights • BMLP architectures • FCC architectures Neural network architectures and learning algorithms, Wilamowski, B.M.

16

Outline

• • • • • Neural Architectures Parity-N Problem Suitable Architectures Use Minimum Network Size Conclusion 17

suitable architectures

• For a limited number of neurons, FCC neural networks are the most powerful architectures, but this does not mean that they are the only suitable architectures 18

suitable architectures

• if the two weights marked by red dotted lines – signal has to be propagated by fewer layers Neural network architectures and learning algorithms, Wilamowski, B.M.

19

Outline

• • • • • Neural Architectures Parity-N Problem Suitable Architectures Use Minimum Network Size Conclusion 20

Use Minimum Network Size

• receive a close-to-optimum answer for all patterns that were never used in training • generalization abilities 21

Case Study

TSK fuzzy controller: (a) Required control surface (b) 8*6 = 48 defuzzification rules TSK fuzzy controller: (a) Trapezoidal membership functions (b) Triangular membership functions Neural network architectures and learning algorithms, Wilamowski, B.M.

22

Case Study

(a) 3 neurons in cascade (12 weights), training error = 0.21049

(b) 4 neurons in cascade (18 weights), training error = 0.049061

(a) 5 neurons in cascade (25 weights), training error = 0.023973

(b) 8 neurons in cascade (52 weights), training error = 1.118E-005 Neural network architectures and learning algorithms, Wilamowski, B.M.

23

time complexity

NBN algorithm can train neural networks 1,000 times faster than the EBP algorithm.

(a) EBP algorithm, average solution time of 4.2s, and average 4188.3 iterations (b) NBN algorithm, average solution time of 2.4ms , and average 5.73 iterations Neural network architectures and learning algorithms, Wilamowski, B.M.

24

two-spiral problem

NBN algorithm using FCC architecture 244 iterations and 0.913s

EBP algorithm using FCC architecture 30,8225 iterations and 342.7s

Neural network architectures and learning algorithms, Wilamowski, B.M.

25

Outline

• • • • • Neural Architectures Parity-N Problem Suitable Architectures Use Minimum Network Size Conclusion 26

Conclusions

• • • FCC or BMLP architectures are not only more powerful but also easier to train use networks with a minimum number of neurons NBN have to invert a nw*nw matrix, but 500 weights are limit now.

27