Transcript Document

Producing Artificial Neural Networks using a Simple Embryogeny
Chris Bowers
School of Computer Science, University of Birmingham
[email protected]
Introduction
Artificial Neural Networks (ANN’s) can be extremely powerful and this power is mainly a
result of the hugely parallel structure exhibited by these systems. However predetermining the
structure of such complex networks is entirely problem dependent and so there is no strong
theory behind how to construct an ANN given a particular problem.
The fact that the structure of a successful ANN cannot be predetermined has resulted in
limited success in producing networks which can solve large complex problems. This is due
to an almost exponential increase in the numbers of parameters describing a network as the
size of a network is increased.
In order to overcome this problem of scalability, neural networks have been integrated into
evolutionary systems (EANN’s) so that optimal neural networks for more complex problems
can be found. This approach has resulted in novel architectures and networks never achieved
by hand coded methods*. However, this has had limited success at producing large scale
complex networks since the size of the genetic search space is directly related to the number
of free parameters in the ANN. So for larger networks this results in an enormous search
space. Therefore, most EANN implementations still result in a lack of scalability when
considering large ANN’s.
Process of mapping genotype to phenotype
The genotypes are mapped to phenotypes using a growth process. This growth process is performed on a grid
which defines an environment where, within each location, a cell can exist. In order to produce a growth step
the state of each cell in the grid is encoded into a binary string upon which the genotype operates. The result of
this operation is the new state which the cell must take. This is done in parallel on each cell in the grid. The
state of a cell consists of:
•the current cell type.
•The cell type of surrounding cells.
•The level of chemical in the surrounding area.
•Axon direction and length
Complex mappings
Figure 2 - Example of individual phenotype
The work shown on this poster is an example of how a simple model of embryogeny can be
used to grow neural networks and identifies some of the difficulties highlighted by this
approach.
White space
Gray
Red
Green
Light green
Yellow
Light blue
Blue
- No Structure/Cell
- Substrate/Dead/Non-functioning cells
- Output neuron
- Hidden neuron
- Developing hidden neuron
- Stem cell
- Developing input neuron
- Output neuron
Black lines
Gray lines
-Axons projected from cells
-Completed axon-dendrite connection between two cells
Figure 1 - An example of the genetic representation
of CGP
0
1
2
3
4
5
6
7
8
0
2
0
5
9
1
4
15
3
18
21
22
1
17
27
28
0
1
6
36
2
5
0
2
10
3
10
14
3
19
19
0
1
4
28
0
1
1
5
37
11
12
14
0
20
8
7
0
29
3
5
12
4
38
19
21
20
17
6
13
30
2
13
21
18
39
18
22
25
12
7
1
31
35
23
0
12
40
16
23
6
19
26
1
32
2
2
23
4
41
9
33
14
17
22
1
42
7
0
3
0
8
2
4
3
12
12
13
1
4
2
2
7
10
13
5
16
0
0
14
8
17
3
7
24
15
4
21
5
2
4
2
6
6
0
11
15
7
13
11
8
2
5
6
16
12
5
0
8
25
17
6
3
5
34
34
22
7
1
43
6
7
0
5
17
1
5
17
4
26
26
4
17
3
35
30
14
8
0
44
Genome
0205250270302431222710524066011825667051415331014312140813141951601881
7316713117125081517421221171901487022017613251271619261154219176352641
732801601153512421321183523012222341417221342271301480
Genotype space
Discussion
Determining the fitness of a genotype
The genotypic structure is based upon a recurrent version of the Cartesian Genetic
Programming (CGP) approach** originally developed for Boolean function learning. The
genotype consists of a set of nodes represented in grid formation (figure 1). The number of
rows in the grid is dependent upon the number of binary outputs required from the Boolean
function. The number of columns is dependent upon the complexity of the Boolean function
required. Each nodes defines a Boolean operation, such as NAND or NOT, and a set of binary
inputs upon which the Boolean function operates. These inputs are either direct inputs to the
CGP or from the output of other nodes. In this manor Boolean operations can be connected
together in a graph to form a Boolean function which operates on a binary string.
Phenotype space
This allows each cell to perform operations such as die, divide, move, differentiate and produce and axon.
In recent years, the use of an even more biologically inspired method for EANN’s has been
suggested. Nature solves this problem of scalability by evolving the rules upon which
biological neural networks are grown. These growth rules are applied at the cellular level
during development in a process known as embryogeny. This means that the size of the
genetic search space is no longer dependent upon the size of the neural networks but upon the
amount of linearity, repetition and modularity exhibited within the resultant network.
Genotype structure
Figure 3 - Visualisation of genotype and phenotype mapping
The fitness of a given genotype is dependent upon the fitness of its resultant phenotype. Since the phenotype
defines a grid occupied by an arrangement of cells with various axons protruding from cells then an
interpretation algorithm is required to produce a ANN structure form this phenotype. The process consists of first
determining each cell type and then determining its connectivity with other cells in the network. Connectivity
between two cells is dependent upon the distance of the axon head of one cell to the body of the other (figure 2).
Once this is completed the network can be evaluated which requires a three stage process.
1. Does the network consist of the correct number of inputs and outputs.
2. Does the network connectivity result in a valid network.
3. How well does the network perform when trained on a set of training patterns taking an average of a number
of randomly initialised networks.
Problems with an embrogeny approach
Since an embryogeny introduces a complex mapping into the evolutionary process this effectively results in two
separate search spaces.
•a genotype search space which consists of all possible growth rules and is entirely dependent upon the
representation used for these growth rules.
•a phenotype search space which consists of all possible neural networks, many of which would produce
identical results and many of which are totally invalid neural networks (this space may be infinite in size).
Considering that the size of the phenotype space is likely to be much larger than that of the genotype space and
that also there is likely to be a many to one mapping between the genotype space and phenotype space means
that the genotype space will only map to certain locations in the phenotype space. Therefore the representation
used will limit the number of possible ANN architectures that can be produced (figure 3). It is imperative in this
case that the representation used exhibit the following two characteristics when considering the search space.
•produce a fitness landscape in the genotypic space which is conducive to evolutionary search, i.e. as smooth
and with as few local optima as possible.
•only map to areas in the phenotype space that consists of valid solutions and to keep neutrality in the fitness
landscape of this areas in the phenotype space to a minimum.
* X. Yao, Evolving artificial neural networks, Proceedings of the IEEE, 87(9):1423-1447, September 1999
** J. F. Miller, P. Thomson, Cartesian Genetic Programming, Third European Conference on Genetic Programming , Edinburgh, April 15-16, 2000, Proceedings published as Lecture Notes in Computer Science, Vol. 1802, pp. 121-132
The aim of this work is not to produce a model of neural cell growth for developmental
biologists or neuroscience but to try to understand the basic forces in development that allow
it to be so powerful in expressing complex systems in comparatively simple genotypes. It is
hoped that in future work it will be shown that development can be used to produce more
complex systems than can currently be produced using a traditional approach without
restrictions from issues of scalability.
Only preliminary testing has been performed in order to determine the capabilities of this
model. Initially a simple XOR network was found quite easily and several solutions have
been found for various sized n-parity problems. However, in order to really test the
scalability of this embryogeny the task was extended to the harder problem of finding a
genome which solves all n-parity problems in n growth steps. This has proved extremely
difficult.
There are a number of important problems which have not been dealt with so far in this
work. Since a simple back propagation method was used to determine the weights in the
network the training performance was still dependent upon the size of the network and so
network size is still limited in this model by computational power required to obtain optimal
weights.
The genotype search space can still be quite large for more complex genotypes and it is
likely that the representation used here results in a rather rugged search space which is
difficult for evolution to navigate.
Further Work
The model described here has highlighted many problem areas in the use of an embryological
system. Mainly that of the huge and rugged genotype landscape which makes evolution
towards an optimal ANN extremely hard. Directions for further work are mainly concerned
with improving the evolvability of the systems described here:
•Ensure the representation used has the flexibility to exhibit safe mutation and allow
development processes to be initiated at different times in the development? In natural
development these are know as canalisation and heterochrony respectively and seem to be
extremely important in allowing efficient evolution.
• The capabilities of a genotype representation can be expanded by adding more genes to the
genotype. However this results in variable genome length which introduces a new set of
problems for evolutionary computation. Nature overcomes these problems using gene
duplication.