Transcript Will an expert system work for my problem?

Lecture 13
Knowledge engineering
Introduction, or what is knowledge
engineering?
 Will an expert system work for my
problem?
 Will a fuzzy expert system work for my
problem?
 Will neural network work for my problem?
 Summary

Slides are based on Negnevitsky, Pearson Education, 2005
1
Rule-based/frame-based expert systems
 Fuzzy Logic
 Neural Networks
 Genetic Algorithms
 Hybrid Intelligent Systems

–
–
–
–
Neuro expert system
Neuro-fuzzy systems
Evolutionary neural systems
……
Slides are based on Negnevitsky, Pearson Education, 2005
2
What is knowledge engineering?
Davis’ law: “For every tool there is a task
perfectly suited to it”.
But…
It would be too optimistic to assume that for
every task there is a tool perfectly suited to it.
We’re going to provide some guidelines for
selecting an appropriate tool for a given task.
Slides are based on Negnevitsky, Pearson Education, 2005
3
The process of building intelligent knowledgebased systems is called knowledge engineering.
Knowledge engineering has six basic phases:
Phase 1: Problem assessment.
Phase 2: Data and knowledge acquisition.
Phase 3: Development of a prototype system.
Phase 4: Development of a complete system.
Phase 5: Evaluation and revision of the system.
Phase 6: Integration and maintenance of the system.
Slides are based on Negnevitsky, Pearson Education, 2005
4
The process of knowledge engineering
P hase 1: P roblem assessm ent
P hase 2: D ata and know ledge acquisition
P hase 3: D evelopm ent of a prototype system
P hase 4: D evelopm ent of a com plete system
P hase 5: E valuation and revision of the system
P hase 6: Integration and m aintenance of the system
Slides are based on Negnevitsky, Pearson Education, 2005
5
Phase 1: Problem assessment
Determine the problem’s characteristics.
 Identify the main participants in the project.
 Specify the project’s objectives.
 Determine the resources needed for building
the system.

Slides are based on Negnevitsky, Pearson Education, 2005
6
Typical problems addressed by intelligent systems
P roblem type
D escription
D iagnosis
Inferring m alfunctions of an object from its behaviour and
recom m ending solutions.
S election
R ecom m ending the best option from a list of possible
alternatives.
P rediction
P redicting the future behaviour of an object from its
behaviour in the past.
C lassification
A ssigning an object to one of the defined classes.
C lustering
D ividing a heterogeneous group of objects into
hom ogeneous subgroups.
O ptim isation
Im proving the quality of solutions until an optim al one is
found.
C ontrol
G overning the behaviour of an object to m eet specified
requirem ents in real-tim e.
Slides are based on Negnevitsky, Pearson Education, 2005
7
Phase 2: Data and knowledge acquisition
Collect and analyse data and knowledge.
 Make key concepts of the system design more
explicit.

Slides are based on Negnevitsky, Pearson Education, 2005
8
The first issue is incompatible data.
Often the data we want to analyse store text in
EBCDIC coding and numbers in packed decimal
format, while the tools we want to use for building
intelligent systems store text in the ASCII code and
numbers as integers with a single- or doubleprecision floating point.
This issue is normally resolved with data transport
tools that automatically produce the code for the
required data transformation.
Slides are based on Negnevitsky, Pearson Education, 2005
9
The second issue is inconsistent data.
Often the same facts are represented differently in
different data bases. If these differences are not
spotted and resolved in time, we might find
ourselves, for example, analysing consumption
patterns of carbonated drinks using data that does not
include Coca-Cola just because it was stored in a
separate database.
Slides are based on Negnevitsky, Pearson Education, 2005
10
The third issue is missing data.
Actual data records often contain blank fields. We
normally would attempt to infer some useful
information from them. In many cases, we can
simply fill the blank fields in with the most common
or average values. In other cases, the fact that a
particular field has not been filled in might itself
provide us with very useful information. For
example, in a job application form, a blank field for
a business phone number might suggest that an
applicant is currently unemployed.
Slides are based on Negnevitsky, Pearson Education, 2005
11
How do we approach knowledge acquisition?



Usually we start with reviewing documents and
reading books, papers and manuals related to the
problem domain.
Once we become familiar with the problem, we
can collect further knowledge through
interviewing the domain expert.
Then we study and analyse the acquired
knowledge, and repeat the entire process again.
Knowledge acquisition is an inherently
iterative process.
Slides are based on Negnevitsky, Pearson Education, 2005
12
Understanding the problem domain is
critical for building intelligent system.
A classical example is given by Donald Michie.
Slides are based on Negnevitsky, Pearson Education, 2005
13
A cheese factory had an experienced cheese-tester
who was approaching retirement age. The factory
manager decided to replace him with an “intelligent
machine”. The human tester tested the cheese by
sticking his finger into a sample and deciding if it
“felt right”. So it was assumed the machine had to
do the same – test for the right surface tension. But
the machine was useless. Eventually, it turned out
that the human tester subconsciously relied on the
cheese’s smell rather than on its surface tension and
used his finger just to break the crust and let the
aroma out.
Slides are based on Negnevitsky, Pearson Education, 2005
14
Phase 3: Development of a prototype system
Choose a tool for building an intelligent system.
 Transform data and represent knowledge.
 Design and implement a prototype system.
 Test the prototype with test cases.

Slides are based on Negnevitsky, Pearson Education, 2005
15
What is a prototype?



A prototype system is defined as a small version of
the final system.
It is designed to test how well we understand the
problem  to make sure that the problem-solving
strategy, the tool selected for building a system, and
techniques for representing acquired data and
knowledge are adequate to the task.
It also provides us with an opportunity to persuade
the sceptics and, in many cases, to actively engage
the domain expert in the system’s development.
Slides are based on Negnevitsky, Pearson Education, 2005
16
What is a test case?


A test case is a problem successfully solved in the
past for which input data and an output solution are
known.
During testing, the system is presented with the
same input data and its solution is compared with
the original solution.
Slides are based on Negnevitsky, Pearson Education, 2005
17
Phase 4: Development of a complete system
Prepare a detailed design for a full-scale system.
 Collect additional data and knowledge.
 Develop the user interface.
 Implement the complete system.

Slides are based on Negnevitsky, Pearson Education, 2005
18
The main work at this phase is often associated with
adding data and knowledge to the system.


If, for example, we develop a diagnostic system,
we might need to provide it with more rules for
handling specific cases.
If we develop a prediction system, we might need
to collect additional historical examples to make
predictions more accurate.
Slides are based on Negnevitsky, Pearson Education, 2005
19
Phase 5: Evaluation and revision of the system
Evaluate the system against the performance
criteria.
 Revise the system as necessary.

Slides are based on Negnevitsky, Pearson Education, 2005
20




Intelligent systems, unlike conventional computer
programs, are designed to solve problems that
quite often do not have clearly defined “right”
and “wrong” solutions.
To evaluate an intelligent system is , in fact, to
assure that the system performs the intended task
to the user’s satisfaction.
A formal evaluation of the system is normally
accomplished with the test cases.
The system’s performance is compared against
the performance criteria that were agreed upon at
the end of the prototyping phase.
Slides are based on Negnevitsky, Pearson Education, 2005
21
Phase 6: Integration and maintenance of
the system
Make arrangements for technology transfer.
 Establish an effective maintenance program.

Slides are based on Negnevitsky, Pearson Education, 2005
22

End of part 1
Slides are based on Negnevitsky, Pearson Education, 2005
23
Will an expert system work for my
problem?
The Phone Call Rule: “Any problem that
can be solved by your in-house expert in a
10-30 minute phone call can be developed
as an expert system”.
Slides are based on Negnevitsky, Pearson Education, 2005
24
Case study 1:
Diagnostic expert system



Diagnostic expert systems are relatively easy to
develop:
Most diagnostic problems have a finite list of
possible solutions,
Involve a rather limited amount of wellformalised knowledge, and
Often take a human expert a short time (say, an
hour) to solve.
Slides are based on Negnevitsky, Pearson Education, 2005
25
Troubleshooting manual for the Macintosh computer
S ection 1: S ystem S tart-u p
P ro b lem
A ctio n
1 .1 . S yste m d o es no t start






1 .2 . S yste m starts a nd the n freezes
 R estart the M ac.
 U n ho o k all S C S I d evices a nd restart the M ac.
 R estart w ith the shift ke y d o w n (turns o ff
extensio ns).
 R e m o ve all e xten sio n s and ad d the m b ac k o ne at a
tim e. R estart the M ac after eac h ad d itio n. If u sin g
S yste m 7 .5 o r higher, u se E xte nsio ns M a na ger.
1 .3 . S yste m starts w ith a S ad M ac
 S tart w ith D isk T o o ls and d o a clean reinstall o f the
syste m .
 R u n D isk F irst A id , M acC hec k, o r N o rto n’s D isk
D o cto r.
 S tart w ith D isk T o o ls, and if the hard d rive ico n
d o es no t sho w up , call A V R ep air.
1 .4 . S yste m starts w ith w ro n g “m u sic”
 C hec k cab les.
C hec k
C hec k
C hec k
C hec k
C hec k
C hec k
p o w er co rd s (b o th end s).
the p o w er strip .
scree n b righ tne ss.
p ho net co n necto rs.
ke yb o ard co nnecto rs.
p in s co n necto rs.
Slides are based on Negnevitsky, Pearson Education, 2005
26
General rule structure
In each rule, we include a clause that identifies the
current task:
R u le : 1
if
ta sk is ‘syste m sta rt-u p ’
th e n a sk p ro b le m
R u le : 2
if
ta sk is ‘syste m sta rt-u p ’
a n d p ro b le m is ‘syste m d o e s n o t sta rt’
th e n a sk ‘te st p o w e r co rd s’
R u le : 3
if
ta sk is ‘syste m sta rt-u p ’
a n d p ro b le m is ‘syste m d o e s n o t sta rt’
a n d ‘te st p o w e r co rd s’ is o k
th e n a sk ‘te st th e p o w e r strip ’
Slides are based on Negnevitsky, Pearson Education, 2005
27
How do we choose an expert system
development tool?



Tools range from high-level programming languages
such as LISP, PROLOG, OPS, C and Java, to expert
system shells.
High-level programming languages offer a greater
flexibility, but they require high-level programming
skills.
Shells provide us with the built-in inference engine,
explanation facilities and the user interface. We do
not need any programming skills to use a shell – we
enter rules in English in the shell’s knowledge base.
Slides are based on Negnevitsky, Pearson Education, 2005
28
How do we choose an expert system shell?





When selecting an expert system shell, we consider
how the shell represents knowledge (rules or frames);
what inference mechanism it uses (forward or
backward chaining);
whether the shell supports inexact reasoning and if so
what technique it uses (Bayesian reasoning, certainty
factors or fuzzy logic);
whether the shell has an “open” architecture allowing
access to external data files and programs;
how the user will interact with the expert system
(graphical user interface, hypertext).
Slides are based on Negnevitsky, Pearson Education, 2005
29
Case study 2:
Classification expert system
Classification problems can be handled well by
both expert systems and neural networks.
As an example, we will build an expert system
to identify different classes of sail boats. We
start with collecting some information about
mast structures and sail plans of different sailing
vessels. Each boat can be uniquely identified by
its sail plans.
Slides are based on Negnevitsky, Pearson Education, 2005
30
Eight classes of sailing vessels
Jib-headed C utter
G aff-headed Sloop
S taysail S chooner
G aff-headed S chooner
Jib-headed Y aw l
G aff-headed Y aw l
Jib-headed K etch
G aff-headed K etch
Slides are based on Negnevitsky, Pearson Education, 2005
31
Rules for the boat classification expert system
/* S a ilin g V e s s e l C la s s ific atio n E x p e rt S ys tem : M a rk 1
R u le : 1
if
‘th e n u m b er o f m a sts ’ is o n e
a n d ‘th e s h a p e o f th e m a in s a il’ is tria n g u la r
th e n b o a t is ‘J ib -h ea d e d C u tte r’
R u le : 2
if
‘th e n u m b er o f m a sts ’ is o n e
a n d ‘th e s h a p e o f th e m a in s a il’ is q u a drila te ra l
th e n b o a t is ‘G a ff-h e a d e d S lo o p ’
R u le : 3
if
and
and
and
th e n
‘th e n um b e r of m a s ts ’ is tw o
‘th e m a in m a s t p o s itio n ’ is ‘fo rw ard of th e s h ort m a s t’
‘th e s h ort m a s t p o s itio n ’ is ‘fo rw a rd of th e h e lm ’
‘th e s h ap e of th e m a in s a il’ is tria n g u la r
b o a t is ‘J ib -h e a d e d K e tc h ’
R u le : 4
if
and
and
and
th e n
‘th e n um b e r of m a s ts ’ is tw o
‘th e m a in m a s t p o s itio n ’ is ‘fo rw ard of th e s h ort m a s t’
‘th e s h ort m a s t p o s itio n ’ is ‘fo rw a rd of th e h e lm ’
‘th e s h ap e of th e m a in s a il’ is q u a d rila te ra l
b o a t is ‘G aff-h e a d e d K e tc h ’
Slides are based on Negnevitsky, Pearson Education, 2005
32
Continued
R u le : 5
if
and
and
and
th e n
‘th e n um b e r of m a s ts ’ is tw o
‘th e m a in m a s t p o s itio n ’ is ‘fo rw ard of th e s h ort m a s t’
‘th e s h ort m a s t p o s itio n ’ is ‘aft th e h e lm ’
‘th e s h ap e of th e m a in s a il’ is tria n g u la r
b o a t is ‘J ib -h e a d e d Y a w l’
R u le : 6
if
and
and
and
th e n
‘th e n um b e r of m a s ts ’ is tw o
‘th e m a in m a s t p o s itio n ’ is ‘fo rw ard of th e s h ort m a s t’
‘th e s h ort m a s t p o s itio n ’ is ‘aft th e h e lm ’
‘th e s h ap e of th e m a in s a il’ is q u a d rila te ra l
b o a t is ‘G aff-h e a d e d Y a w l’
R u le : 7
if
and
and
th e n
‘th e n um b e r of m a s ts ’ is tw o
‘th e m a in m a s t p o s itio n ’ is ‘a ft th e s h ort m a st’
‘th e s h ap e of th e m a in s a il’ is q u a d rila te ra l
b o a t is ‘G aff-h e a d e d S c h o o n e r’
R u le : 8
if
and
and
th e n
‘th e n um b e r of m a s ts ’ is tw o
‘th e m a in m a s t p o s itio n ’ is ‘a ft th e s h ort m a st’
‘th e s h ap e of th e m a in s a il’ is ‘tria n g u la r w ith tw o fo re s a ils ’
b o a t is ‘S ta y s a il S c h o o n er’
/***************************************************************************
/* T h e S E E K d ire ctiv e s e ts u p th e g o a l
s e e k b o at
Slides are based on Negnevitsky, Pearson Education, 2005
33
Solving classification problems with
certainty factors
Although solving real-world classification
problems often involves inexact and incomplete
data, we still can use the expert system
approach.
However, we need to deal with uncertainties.
The certainty factors theory can manage
incrementally acquired evidence, as well as
information with different degrees of belief.
Slides are based on Negnevitsky, Pearson Education, 2005
34
Uncertainty management in the boat
classification expert system
/* S a ilin g V e s s e l C la s s ific atio n E x p e rt S ys tem : M a rk 2
c o n tro l c f
if
th e n
‘th e n u m b er o f m a sts ’ is o n e
b o a t is ‘J ib -h ea d e d C u tte r’
b o a t is ‘G a ff-h e a d e d S lo o p ’
R u le : 2
if
and
th e n
‘th e n u m b er o f m a sts ’ is o n e
‘th e s h a p e o f th e m a in s a il’ is tria n g u la r
b o a t is ‘J ib -h ea d e d C u tte r’
{c f 1 .0}
R u le : 3
if
and
th e n
‘th e n u m b er o f m a sts ’ is o n e
‘th e s h a p e o f th e m a in s a il’ is q u a drila te ra l
b o a t is ‘G a ff-h e a d e d S lo o p ’
{c f 1 .0}
R u le : 4
if
th e n
‘th e n u m b er o f m a sts ’ is tw o
b o a t is ‘J ib -h ea d e d K etc h ’
b o a t is ‘G a ff-h e a d e d K e tch ’
b o a t is ‘J ib -h ea d e d Y a w l’
b o a t is ‘G a ff-h e a d e d Y a w l’
b o a t is ‘G a ff-h e a d e d S c h o o n er’
b o a t is ‘S tay s a il S c h o o n e r’
R u le : 1
Slides are based on Negnevitsky, Pearson Education, 2005
{c f 0 .4};
{c f 0 .4}
{c f
{c f
{c f
{c f
{c f
{c f
0 .1};
0 .1};
0 .1};
0 .1};
0 .1};
0 .1}
35
Continued
R u le : 5
if
and
th e n
‘n u m b e r of m a sts ’ is tw o
‘m a in m a st p os itio n ’ is ‘fo rw ard o f th e s h o rt m a st’
b o a t is ‘J ib -h ea d e d K etc h ’
{c f 0 .2};
b o a t is ‘G a ff-h e a d e d K e tch ’
{c f 0 .2};
b o a t is ‘J ib -h ea d e d Y a w l’
{c f 0 .2};
b o a t is ‘G a ff-h e a d e d Y a w l’
{c f 0 .2}
R u le : 6
if
and
th e n
‘th e n u m b er o f m a sts ’ is tw o
‘th e m a in m a st po s itio n ’ is ‘a ft th e s h o rt m a s t’
b o a t is ‘G a ff-h e a d e d S c h o o n er’
{c f 0 .4};
b o a t is ‘S tay s a il S c h o o n e r’
{c f 0 .4}
R u le : 7
if
and
th e n
‘th e n u m b er o f m a sts ’ is tw o
‘th e s h o rt m a st p os itio n ’ is ‘forw a rd of th e h e lm ’
b o a t is ‘J ib -h ea d e d K etc h ’
{c f 0 .4};
b o a t is ‘G a ff-h e a d e d K e tch ’
{c f 0 .4}
R u le : 8
if
and
th e n
‘th e n u m b er o f m a sts ’ is tw o
‘th e s h o rt m a st p os itio n ’ is ‘a ft th e h e l m ’
b o a t is ‘J ib -h ea d e d Y a w l’
{c f
b o a t is ‘G a ff-h e a d e d Y a w l’
{c f
b o a t is ‘G a ff-h e a d e d S c h o o n er’
{c f
b o a t is ‘S tay s a il S c h o o n e r’
{c f
if
‘th e n u m b er o f m a sts ’ is tw o
R u le : 9
Slides are based on Negnevitsky, Pearson Education, 2005
0 .2};
0 .2};
0 .2};
0 .2}
36
Continued
R u le : 9
if
and
th e n
‘th e n u m b er o f m a sts ’ is tw o
‘th e s h a p e o f th e m a in s a il’ is tria n g u la r
b o a t is ‘J ib -h ea d e d K etc h ’
{c f 0 .4};
b o a t is ‘J ib -h ea d e d Y a w l’
{c f 0 .4}
R u le : 1 0
if
and
th e n
‘th e n u m b er o f m a sts ’ is tw o
‘th e s h a p e o f th e m a in s a il’ is q u a drila te ra l
b o a t is ‘G a ff-h e a d e d K e tch ’
{c f 0 .3};
b o a t is ‘G a ff-h e a d e d Y a w l’
{c f 0 .3};
b o a t is ‘G a ff-h e a d e d S c h o o n er’
{c f 0 .3}
R u le : 1 1
if
and
th e n
‘th e n u m b er o f m a sts ’ is tw o
‘th e s h a p e o f th e m a in s a il’ is ‘tria n g u la r w ith tw o fo re s a ils ’
b o a t is ‘S tay s a il S c h o o n e r’
{c f 1 .0}
s e e k b o at
Slides are based on Negnevitsky, Pearson Education, 2005
37
Will a fuzzy expert system work for
my problem?
If you cannot define a set of exact rules for each
possible situation, then use fuzzy logic.
While certainty factors and Bayesian probabilities
are concerned with the imprecision associated with
the outcome of a well-defined event, fuzzy logic
concentrates on the imprecision of the event itself.
Inherently imprecise properties of the problem
make it a good candidate for fuzzy technology.
Slides are based on Negnevitsky, Pearson Education, 2005
38
Case study 3:
Decision-support fuzzy systems
Although, most fuzzy technology applications
are still reported in control and engineering, an
even larger potential exists in business and
finance. Decisions in these areas are often based
on human intuition, common sense and
experience, rather than on the availability and
precision of data.
Fuzzy technology provides us with a means of
coping with the “soft criteria” and “fuzzy data”
that are often used in business and finance.
Slides are based on Negnevitsky, Pearson Education, 2005
39
Mortgage application assessment is a typical
problem to which decision-support fuzzy
systems can be successfully applied.
Assessment of a mortgage application is
normally based on evaluating the market value
and location of the house, the applicant’s assets
and income, and the repayment plan, which is
decided by the applicant’s income and bank’s
interest charges.
Slides are based on Negnevitsky, Pearson Education, 2005
40
Fuzzy sets of the linguistic variable Market value
D egree of
M em bership
1.0
0.8
L ow
M edium
V ery high
H igh
0.6
0.4
0.2
0.0
0
100
200
300
400
500
600
Slides are based on Negnevitsky, Pearson Education, 2005
700
800
900 1,000  10 3
M arket value, $
41
Fuzzy sets of the linguistic variable Location
D egree of
M em bership
1.0
0.8
B ad
E xcellent
F air
0.6
0.4
0.2
0.0
0
1
2
3
4
5
6
Slides are based on Negnevitsky, Pearson Education, 2005
7
8
9
10
L ocation
42
Fuzzy sets of the linguistic variable House
D egree of
M em bership
1.0
0.8
V ery low
L ow
M edium
H igh
V ery high
0.6
0.4
0.2
0.0
0
1
2
3
4
5
6
Slides are based on Negnevitsky, Pearson Education, 2005
7
8
9
10
H ouse
43
Fuzzy sets of the linguistic variable Asset
D egree of
M em bership
1.0
0.8
L ow
H igh
M edium
0.6
0.4
0.2
0.0
0
100
200
300
400
500
600
Slides are based on Negnevitsky, Pearson Education, 2005
700
800
900 1,000  10 3
A sset, $
44
Fuzzy sets of the linguistic variable Income
D egree of
M em bership
1.0
0.8
0.6
L ow
M edium
H igh
V ery high
0.4
0.2
0.0
0
10
20
30
40
50
60
Slides are based on Negnevitsky, Pearson Education, 2005
70
80
100  10 3
90
Incom e, $
45
Fuzzy sets of the linguistic variable Applicant
D egree of
M em bership
1.0
0.8
L ow
M edium
H igh
0.6
0.4
0.2
0.0
0
1
2
3
4
5
6
Slides are based on Negnevitsky, Pearson Education, 2005
7
8
9
10
A pplicant
46
Fuzzy sets of the linguistic variable Interest
D egree of
M em bership
1.0
0.8
L ow
H igh
M edium
0.6
0.4
0.2
0.0
0
1
2
3
4
5
6
Slides are based on Negnevitsky, Pearson Education, 2005
7
8
9
10
Interest, %
47
Fuzzy sets of the linguistic variable Credit
D egree of
M em bership
1.0
0.8
V ery low
0.6
L ow
M edium
H igh
V ery high
0.4
0.2
0.0
0
50
100
150
200
250
300
Slides are based on Negnevitsky, Pearson Education, 2005
350
400
500  10 3
450
C redit, $
48
Rules for mortgage loan assessment
R u le B a s e 1: H o m e E v a lu a tio n
1 . If
2 . If
3 . If
4 . If
5 . If
6 . If
7 . If
8 . If
9 . If
1 0 . If
1 1 . If
1 2 . If
1 3 . If
1 4 . If
(M a rk e t_ v a lu e is L o w ) th e n (H o u s e is L o w )
(L o c a tio n is B a d ) th e n (H o u s e is L o w )
(L o c a tio n is B a d ) a n d (M a rk e t_ v a lu e is L o w ) th e n (H o u s e is V e ry _ lo w )
(L o c a tio n is B a d ) a n d (M a rk e t_ v a lu e is M e d iu m ) th e n (H o u s e is L o w )
(L o c a tio n is B a d ) a n d (M a rk e t_ v a lu e is H ig h ) th e n (H o u s e is M e d iu m )
(L o c a tio n is B a d ) a n d (M a rk e t_ v a lu e is V e ry _ h ig h ) th e n (H o u s e is H ig h )
(L o c a tio n is F a ir) a n d ( M a rk e t_ v a lu e is L o w ) th e n (H o u s e is L o w )
(L o c a tio n is F a ir) a n d ( M a rk e t_ v a lu e is M e d iu m ) th e n (H o u s e is M e d iu m )
(L o c a tio n is F a ir) a n d ( M a rk e t_ v a lu e is H ig h ) th e n (H o u s e is H ig h )
(L o c a tio n is F a ir) a n d ( M a rk e t_ v a lu e is V e ry _ h ig h ) th e n (H o u s e is V e ry _ h ig h )
(L o c a tio n is E x c e lle n t) a n d ( M a rk e t_ v a lu e is L o w ) th e n (H o u s e is M e d iu m )
(L o c a tio n is E x c e lle n t) a n d ( M a rk e t_ v a lu e is M e d iu m ) th e n (H o u s e is H ig h )
(L o c a tio n is E x c e lle n t) a n d ( M a rk e t_ v a lu e is H ig h ) th e n (H o u s e is V e ry _ h ig h )
(L o c a tio n is E x c e lle n t) a n d ( M a rk e t_ v a lu e is V e ry _ h ig h ) th e n (H o u s e is V e ry _ h ig h )
Slides are based on Negnevitsky, Pearson Education, 2005
49
Rules for mortgage loan assessment
R u le B a s e 2: A p p lic a n t E v alu a tio n
1 . If
2 . If
3 . If
4 . If
5 . If
6 . If
7 . If
8 . If
9 . If
1 0 . If
1 1 . If
1 2 . If
(A s s e t
(A s s e t
(A s s e t
(A s s e t
(A s s e t
(A s s e t
(A s s e t
(A s s e t
(A s s e t
(A s s e t
(A s s e t
(A s s e t
is
is
is
is
is
is
is
is
is
is
is
is
L o w ) a n d (In c o m e is L o w ) th e n (A p p lic a n t is L o w )
L o w ) a n d (In c o m e is M e d iu m ) th e n (A p p lic a n t is L o w )
L o w ) a n d (In c o m e is H ig h ) th e n (A p p lic a n t is M e d iu m )
L o w ) a n d (In c o m e is V e ry _ h ig h ) th e n (A p p lic a n t is H ig h )
M e d iu m ) a n d (In c o m e is L o w ) th e n (A p p lic a n t is L o w )
M e d iu m ) a n d (In c o m e is M e d iu m ) th e n (A p p lic a n t is M e d iu m )
M e d iu m ) a n d (In c o m e is H ig h ) th e n (A p p lic a n t is H ig h )
M e d iu m ) a n d (In c o m e is V e ry _ h ig h ) th e n (A p p lic a n t is H ig h )
H ig h ) a n d (In c o m e is L o w ) th e n (A p p lic a n t is M e d iu m )
H ig h ) a n d (In c o m e is M e d iu m ) th e n (A p p lic a n t is M e d iu m )
H ig h ) a n d (In c o m e is H ig h ) th e n (A p p lic a n t is H ig h )
H ig h ) a n d (In c o m e is V e ry _ h ig h ) th e n (A p p lic a n t is H ig h )
Slides are based on Negnevitsky, Pearson Education, 2005
50
Rules for mortgage loan assessment
R u le B a s e 3: C re d it E v a lu atio n
1 . If
2 . If
2 . If
4 . If
5 . If
6 . If
7 . If
8 . If
9 . If
1 0 . If
1 1 . If
1 2 . If
1 3 . If
1 4 . If
1 5 . If
(In c o m e is L o w ) a n d (In te re s t is M e d iu m ) th e n (C re d it is V e ry _ lo w )
(In c o m e is L o w ) a n d (In te re s t is H ig h ) th e n (C re d it is V e ry _ lo w )
(In c o m e is M e d iu m ) a n d (In te re s t is H ig h ) th e n (C re d it is L o w )
(A p p lic a n t is L o w ) th e n (C re d it is V e ry _ lo w )
(H o u s e is V e ry _ lo w ) th e n (C re d it is V e ry _ lo w )
(A p p lic a n t is M e d iu m ) a n d (H o u s e is V e ry _ lo w ) th e n (C re d it is L o w )
(A p p lic a n t is M e d iu m ) a n d (H o u s e is L o w ) th e n (C re d it is L o w )
(A p p lic a n t is M e d iu m ) a n d (H o u s e is M e d iu m ) th e n (C re d it is M e d iu m )
(A p p lic a n t is M e d iu m ) a n d (H o u s e is H ig h ) th e n (C re d it is H ig h )
(A p p lic a n t is M e d iu m ) a n d (H o u s e is V e ry _ h ig h ) th e n (C re d it is H ig h )
(A p p lic a n t is H ig h ) a n d (H o u s e is V e ry _ lo w ) th e n (C re d it is L o w )
(A p p lic a n t is H ig h ) a n d (H o u s e is L o w ) th e n (C re d it is M e d iu m )
(A p p lic a n t is H ig h ) a n d (H o u s e is M e d iu m ) th e n (C re d it is H ig h )
(A p p lic a n t is H ig h ) a n d ( H o u s e is H ig h ) th e n (C re d it is H ig h )
(A p p lic a n t is H ig h ) a n d (H o u s e is V e ry _ h ig h ) th e n (C re d it is V e ry _ h ig h )
Slides are based on Negnevitsky, Pearson Education, 2005
51
Hierarchical fuzzy model
M arket value
L ocation
A sset
R u le B a s e 1 : H o m e E v a lu a tio n
H ouse
Incom e
R u le B a s e 2 : A p p lic a n t E v a lu a tio n
Interest
Incom e
A pplicant
R u le B a s e 3 : C re d it E v a lu a tio n
C redit
Slides are based on Negnevitsky, Pearson Education, 2005
52
Three-dimensional plots for Rule Base 1
and Rule base 2
9
8
A p p lic a nt
8
House
7
6
5
7
6
5
4
4
3
3
2
1 ,0 0 0 ,0 0 0
1 0 0 ,0 0 0
6 0 0 ,0 0 0
2 0 0 ,0 0 0
M a rk e t v alu e , $
0
2
4
6
L o c a tio n
8
1 ,0 0 0 ,0 0 0
10
6 0 ,0 0 0
In c o m e, $
(a)
Slides are based on Negnevitsky, Pearson Education, 2005
6 0 0 ,0 0 0
2 0 ,0 0 0
0
2 0 0 ,0 0 0
A ss e t, $
(b)
53
Three-dimensional plots for Rule Base 3
3 0 0 ,0 0 0
C re d it, $
C re d it, $
4 0 0 ,0 0 0
3 0 0 ,0 0 0
2 0 0 ,0 0 0
2 5 0 ,0 0 0
2 0 0 ,0 0 0
1 5 0 ,0 0 0
1 0 0 ,0 0 0
1 0 0 ,0 0 0
5 0 ,0 0 0
10
10
5
House
0
0
2
4
6
A p p lic a nt
8
10
8
5
0
A p p lic a nt
(a)
Slides are based on Negnevitsky, Pearson Education, 2005
0
2
4
10
6
In te re st, %
(b)
54
Will a neural network work for my
problem?
Neural networks represent a class of very
powerful, general-purpose tools that have been
successfully applied to prediction, classification
and clustering problems. They are used in a
variety of areas, from speech and character
recognition to detecting fraudulent transactions,
from medical diagnosis of heart attacks to process
control and robotics, from predicting foreign
exchange rates to detecting and identifying radar
targets.
Slides are based on Negnevitsky, Pearson Education, 2005
55
Case study 4:
Character recognition Neural networks
Recognition of both printed and handwritten
characters is a typical domain where neural
networks have been successfully applied.
Optical character recognition systems were
among the first commercial applications of
neural networks.
Slides are based on Negnevitsky, Pearson Education, 2005
56
We demonstrate an application of a multilayer
feedforward network for printed character
recognition.
For simplicity, we can limit our task to the
recognition of digits from 0 to 9. Each digit is
represented by a 5  9 bit map.
In commercial applications, where a better
resolution is required, at least 16  16 bit maps
are used.
Slides are based on Negnevitsky, Pearson Education, 2005
57
Bit maps for digit recognition
1
2
3
4
5
6
7
8
9
10
11 12 13 14 15
16 17 18 19 20
21 22 23 24 25
26 27 28 29 30
31 32 33 34 35
36 37 38 39 40
41 42 43 44 45
Slides are based on Negnevitsky, Pearson Education, 2005
58
How do we choose the architecture of a
neural network?


The number of neurons in the input layer is
decided by the number of pixels in the bit map.
The bit map in our example consists of 45
pixels, and thus we need 45 input neurons.
The output layer has 10 neurons – one neuron
for each digit to be recognised.
Slides are based on Negnevitsky, Pearson Education, 2005
59
How do we determine an optimal number
of hidden neurons?


Complex patterns cannot be detected by a small
number of hidden neurons; however too many
of them can dramatically increase the
computational burden.
Another problem is overfitting. The greater the
number of hidden neurons, the greater the
ability of the network to recognise existing
patterns. However, if the number of hidden
neurons is too big, the network might simply
memorise all training examples.
Slides are based on Negnevitsky, Pearson Education, 2005
60
Neural network for printed digit recognition
0
1
1
0
1
2
2
1
1
3
3
0
4
0
5
0
6
0
7
0
8
0
1
0
4
5
1
2
3
1
41
1
42
1
43
1
44
9
0
1
45
10
0
4
5
Slides are based on Negnevitsky, Pearson Education, 2005
61
What are the test examples for character
recognition?



A test set has to be strictly independent from the
training examples.
To test the character recognition network, we
present it with examples that include “noise” –
the distortion of the input patterns.
We evaluate the performance of the printed digit
recognition networks with 1000 test examples
(100 for each digit to be recognised).
Slides are based on Negnevitsky, Pearson Education, 2005
62
Learning curves of the digit recognition
three-layer neural networks
10
S u m -S q u a re d E rro r
10
1
1
1
1
1
2
1
1
0
4
-1
1
2
3
4
-2
–
–
–
–
3
2
tw o h id d e n n e u ro n s;
five h id d e n n e u ro n s;
te n h id d e n n e u ro n s;
tw e n ty h id d e n n e u ro n s
-3
-4
0
50
100
150
E p o ch s
Slides are based on Negnevitsky, Pearson Education, 2005
200
250
63
Performance evaluation of the digit
recognition neural networks
25
R e co g n itio n E rro r, %
20
15
2 – five h id d e n n e u ro n s;
3 – te n h id d e n n e u ro n s;
4 – tw e n ty h id d e n n e u ro n s
10
2
3
4
5
0
0
0 .1
0 .2
0 .3
N o ise L e ve l
Slides are based on Negnevitsky, Pearson Education, 2005
0 .4
0 .5
64
Can we improve the performance of the
character recognition neural network?
A neural network is as good as the examples
used to train it.
Therefore, we can attempt to improve digit
recognition by feeding the network with
“noisy” examples of digits from 0 to 9.
Slides are based on Negnevitsky, Pearson Education, 2005
65
Performance evaluation of the digit recognition
network trained with “noisy” examples
16
14
n e tw o rk tra in e d w ith “p e rfe ct” e xa m p le s
n e tw o rk tra in e d w ith “n o isy” e xa m p le s
R e co g n itio n E rro r, %
12
10
8
6
4
2
0
0
0 .0 5
0 .1
0 .1 5
0 .2
0 .2 5
0 .3
0 .3 5
0 .4
0 .4 5
0 .5
N o ise L e ve l
Slides are based on Negnevitsky, Pearson Education, 2005
66
Case study 5:
Prediction neural networks
As an example, we consider a problem of
predicting the market value of a given house
based on the knowledge of the sales prices of
similar houses.
Slides are based on Negnevitsky, Pearson Education, 2005
67



In this problem, the inputs (the house location,
living area, number of bedrooms, number of
bathrooms, land size, type of heating system,
etc.) are well-defined, and even standardised for
sharing the housing market information between
different real estate agencies.
The output is also well-defined – we know what
we are trying to predict.
The features of recently sold houses and their
sales prices are examples, which we use for
training the neural network.
Slides are based on Negnevitsky, Pearson Education, 2005
68
Network generalisation
An appropriate number of training examples can
be estimated with Widrow’s rule of thumb, which
suggests that, for a good generalisation, we need to
satisfy the following condition:
N 
nw
e
where N is the number of training examples, nw is
the number of synaptic weights in the network, and
e is the network error permitted on test.
Slides are based on Negnevitsky, Pearson Education, 2005
69
Massaging the data

Data can be divided into three main types:
continuous, discrete and categorical .
Continuous data vary between two pre-set
values – minimum and maximum, and can be
mapped, or massaged, to the range between 0
and 1 as:
massaged
value 
actual value - minimum value
maximum value - minimum value
Slides are based on Negnevitsky, Pearson Education, 2005
70

Discrete data, such as the number of bedrooms
and the number of bathrooms, also have
maximum and minimum values. For example, the
number of bedrooms usually ranges from 0 to 4.
Massaging the data
0
0
no bedroom s
0 .2 5
one bedroom
0 .5 0
tw o bedroom s
0 .7 5
three bedroom s
1
four or m ore bedroom s
1
N um ber of bedroom s 2
3
4
Slides are based on Negnevitsky, Pearson Education, 2005
71

Categorical data, such as gender and marital
status, can be massaged by using 1 of N coding.
This method implies that each categorical value is
handled as a separate input.
For example, marital status, which can be either
single, divorced, married or widowed, would be
represented by four inputs. Each of these inputs
can have a value of either 0 or 1. Thus, a married
person would be represented by an input vector
[0 0 1 0].
Slides are based on Negnevitsky, Pearson Education, 2005
72
Feedforward neural network for
real-estate appraisal
L o c atio n ra tin g
7
0 .7 0 00
1
H o u s e c o nd itio n
8
0 .8 0 00
2
N u m b e r o f be d ro om s
3
0 .7 5 00
3
N u m b e r o f ba th roo m s
1
0 .5 0 00
4
L ivin g a re a
1 2 1 .00 m
2
0 .3 5 50
5
L a n d s ize
6 8 2 .00 m
2
0 .4 6 82
6
G a ra g e
d o u b le
1 .0 0 00
7
1 .0 0 00
8
0 .0 0 00
9
0 .0 0 00
10
T yp e o f h e a tin g s ys tem w o o d h e a te r
Slides are based on Negnevitsky, Pearson Education, 2005
0.3546
$ 1 2 0 ,9 2 0
73
How do we validate results?
To validate results, we use a set of examples
never seen by the network.
Before training, all the available data are
randomly divided into a training set and a test set.
Once the training phase is complete, the
network’s ability to generalise is tested against
examples of the test set.
Slides are based on Negnevitsky, Pearson Education, 2005
74