Transcript Machine Learning

Machine Learning
Concept Learning
General-to Specific Ordering
(Inductive Classification)
Note
• Simple approach
– Assumption: no noise
– Illustrates key concepts
Concept learning task
• Concept learning = classification (categorizatin)
• Given examples, learn a general concept (category)
– subset of some general domain
– boolean-valued function over the domain (characteristic function)
• Determine
– infer a boolean-valued function from samples of it (input, output)
Training Examples for EnjoySport
• Concept learning: inferring a boolean-valued function
from training examples
• Binary classification
• Concept to learn: Enjoy Sport
Sky
Temp
Humid
Wind
Water
Fore
Cast
Enjoy
Sport
Sunny
Warm
Normal
Strong
Warm
Same
Yes
Sunny
Warm
High
Strong
Warm
Same
Yes
Rainy
Cold
High
Strong
Warm
Change
No
Sunny
Warm
High
Strong
Cool
Change
No
Representing Hypotheses
• Many possible representations
• Here, h is conjunction of constraints on attributes
• Each constraint can be
– a specific value (e.g., Water=Warm)
– don't care (e.g., Water=?)
– no value allowed (e.g., Water=Ø)
• For example,
<Sky Temp
<Sunny ?
Humid Wind Water Forecast >
?
Strong ?
Same >
Concept learning task
• Given:
– A description of an instance, xX, where X is the instance
language or instance space.
– A fixed set of categories: C={c1, c2,…cn}
• Determine:
– The category of x: c(x)C, where c(x) is a categorization function
whose domain is X and whose range is C.
– If c(x) is a binary function C={0,1} ({true,false}, {positive,
negative}, {yes, no}) then it is called a concept.
Example task
• “Days on which Aldo enjoys his water sports”
– day = set of attributes
– predict the value of one attribute given values of others
• Representation?
– Conjunction of constraints on attributes
Sky
Temp
Humid
Wind
Water
Fore
Cast
Enjoy
Sport
Sunny
Warm
Normal
Strong
Warm
Same
Yes
Sunny
Warm
High
Strong
Warm
Same
Yes
Rainy
Cold
High
Strong
Warm
Change
No
Sunny
Warm
High
Strong
Cool
Change
No
Notation
• X: set of items over which the concept is defined
• c: target concept
– function X  {0,1}
– c(x) = 1: x is a positive example of c
– c(x) = 0: x is a negative example of c
• D: set of examples <x, c(x)>
• H: hypothesis space
– design choice
– members of H are functions X -> 0/1
• Goal of (concept) learning
– find h in H such that h(x) = c(x) for all x in X
Prototypical Concept Learning Task
• Given:
– Instances X: Possible days, each described by the attributes
Sky, Temp, Humidity, Wind, Water, Forecast
– Target function c: EnjoySport: X  {0,1}
– Hypotheses H: Conjunctions of literals. E.g.
< ?, Cold, High, ?, ?, ?>
– Training examples D: Positive and negative examples of the
target function
<x1, c(x1)> , … <xn, c(xn)>
• Determine: A hypothesis h in H such that h(x)=c(x) for all
x in D.
Inductive Learning Hypothesis
Any hypothesis found to approximate the
target function well over a sufficiently large
set of training examples will also
approximate the target function well over
other unobserved examples.
Notes
• Unique
– most general hypothesis
– most specific hypothesis
• Concept learning task requires
–
–
–
–
domain (set of instances)
target function
set of candidate hypotheses
set of (labeled) examples
Induction Learning hypothesis
• All we know of c are the examples
• Best we can do is to produce a h consistent with
example data
• Hypotheses
– best h regarding unseen instances is the best h
regarding seen ones
– fundamental assumption in inductive learning
Concept learning as search
• H = search space
– find h best fitting to D
• Selection of representation defines search
space
– size of space (syntactic/semantic)
• Efficient search in very large (or infinite) spaces
for best h
General-to-Specific order
• Useful structure
– organize the search process
– exists for any concept learning task
– search possible without enumerating all members of
H (may be infinite)
• h1 ‘more general than or equal’ h2
– for all x: h1(x) = 1 implies h2(x) = 1
– independent of c!
– defines a partial order on H
Instance, Hypotheses, and MoreGeneral-Than
Find-S Algorithm
1.
Initialize h to the most specific hypothesis in H
2.
For each positive training instance x
For each attribute constraint ai in h
If the constraint ai in h is satisfied by x
Then do nothing
Else replace ai in h by the next more general constraint that is
satisfied by x
3.
Output hypothesis h
Find-S Algorithm
Key property of Find-S
• For H defined as conjunction of constraints
– finds the most specific h consistent with the positive
examples
– consistent with negative ones if no noise and c is in H
Problems of Find-S
• Can not tell whether has learned c
– have we converged to c?
– if not, how uncertain we are of c?
• Why choose most specific h?
• Is training data consistent?
– noise can severely mislead Find-S
– detection, overcoming
• ‘most specific’ is not always unique (Depending on H,
there might be several!)
Version Spaces and Candidate
Elimination
• Another learning approach:
– outputs a description of all members of H that are
consistent with D
– possible without enumerating members of H
(ordering!)
– suffers from noise
– useful framework for introducing many fundamental
issues of ML
Version Space
• Given an hypothesis space, H, and training data, D, the
version space (VS) is the complete subset of H that is
consistent with D.
• The version space can be naively generated for any finite
H by enumerating all hypotheses and eliminating the
inconsistent ones.
VS H ,D  h  H | Consistent(h, D)
Version Space with S and G
• The version space can be represented more compactly
by maintaining two boundary sets of hypotheses, S, the
set of most specific consistent hypotheses, and G, the
set of most general consistent hypotheses:
S  {s  H | Consistent(s, D)  s  H [s  s  Consistent(s, D)]}
G  {g  H | Consistent( g , D)  g   H [ g   g  Consistent(s, D)]}
• S and G represent the entire version space via its
boundaries in the generalization lattice:
G
version
space
S
List-Then-Eliminate Algorithm
• VersionSpace  a list containing every hypothesis in H
• For each training example, <x, c(x)>
remove from VersionSpace any hypothesis h
for which h(x) ≠ c(x)
• Output the list of hypotheses in VersionSpace
Example Version Space
Sky
Temp
Humid
Wind
Water
Fore
Cast
Enjoy
Sport
Sunny
Warm
Normal
Strong
Warm
Same
Yes
Sunny
Warm
High
Strong
Warm
Same
Yes
Rainy
Cold
High
Strong
Warm
Change
No
Sunny
Warm
High
Strong
Cool
Change
No
Representing Version Spaces
• The General boundary, G, of version space VSH,D is
the set of its maximally general members
• The Specific boundary, S, of version space VSH,D is
the set of its maximally specific members
• Every member of the version space lies between
these boundaries
VS H ,D  h  H | (s  S )(g  G)(g  h  s)
where x ≥ y means x is more general or equal to y
Elimination algorithms
• List-then-Eliminate
– something to start with
– ineffective
• Candidate elimination
– same principle
– more compact representation
– maintain most specific and most general elements of
VS(H,D)
Candidate Elimination Algorithm
• G  maximally general hypotheses in H
• S  maximally specific hypotheses in H
• For each training example d, do
If d is a positive example
• Remove from G any hypothesis inconsistent with d
• For each hypothesis s in S that is not consistent with d
– Remove s from S
– Add to S all minimal generalizations h of s such that
1. h is consistent with d, and
2. some member of G is more general than h
– Remove from S any hypothesis that is more general than another hypothesis in S
If d is a negative example
• Remove from S any hypothesis inconsistent with d
• For each hypothesis g in G that is not consistent with d
– Remove g from G
– Add to G all minimal specializations h of g such that
1. h is consistent with d, and
2. some member of S is more specific than h
– Remove from G any hypothesis that is less general than another hypothesis in G
Example Trace
Next training example?
Sky
Temp
Humid
Wind
Water
Fore
Cast
Enjoy
Sport
Sunny
Warm
Normal
Strong
Warm
Same
Yes
Sunny
Warm
High
Strong
Warm
Same
Yes
Rainy
Cold
High
Strong
Warm
Change
No
Sunny
Warm
High
Strong
Cool
Change
No
Sky
Temp
Humid
Wind
Water
Fore
Cast
Enjoy
Sport
Sunny
Warm
Normal
Strong
Warm
Same
Yes
Sunny
Warm
High
Strong
Warm
Same
Yes
Rainy
Cold
High
Strong
Warm
Change
No
Sunny
Warm
High
Strong
Cool
Change
No
How Should These Be Classified?
What Justifies this Inductive Leap?
Remarks on VS & CE
• Converges to correct h?
– no errors, H rich enough: yes
– exact: G = S and both singletons
• Effect of errors (noise)
– example 2 negative  CE removes the correct target
from VS!
– detection: VS gets empty
– similarly when c can not be represented (e.g.
disjunctions)
What example next?
• Assume learner may ask
– analogy: experiments, teacher
– query: instance constructed by learner,
classified by teacher
• What to ask in example case?
– Is there a good general strategy?
– Should attempt to discriminate among
alternatives in VS
What to ask...
• Discriminating example
– c(x) = 1: S will generalize
– c(x) = 0: G will specialize
– optimal choice halves VS
– x satisfies half of VS members
– not possible to generate one in general
How to use partially learned
concepts?
• Classification with ambiguous VS
–
–
–
–
–
–
h(x) = 0/1 for every h in VS: ok
enough to check with G (0) & S (1)
3rd example: 50% support for both
4th: 66% support for 0
majority vote + confidence?
Ok if all h are equally likely
Inductive Bias
• What if c is not in H?
– Use H that includes ‘everything’?
– Will be quite large --> effect on learning
(generalization, # of examples)
• We concentrate us on C-E, but
– results apply to any algorithm outputting any
consistent h for D
Biased hypothesis space
• Assure H contains c
– make H general enough
– what if it’s not?
• Example case
– max specific h consistent with x1 & x2 (+) is
too general for x3 (-)
– reason: we have biased the learner to
consider only conjunctions