Transcript lecture3

Today’s Topics
• HW0 due 11:55pm tonight and no later than next Tuesday
• HW1 out on class home page; discussion page in Moodle
• Please do not use company-specific discussion forums for cs540
– use Moodle (or request we add Piazza)
• Everyone gets 10 free late days this term (but at most 5 per HW)
• Learning from Labeled Examples
• Supervised Learning and Venn Diagrams
• Simple ML Algo: k-Nearest Neighbors
– Read Section 18.8.1 of textbook and Wikipedia article(s) linked
to class home page
• Tuning Parameters
• Some “ML Commandments”
9/15/15
CS 540 - Fall 2015 (© Jude Shavlik), Lecture 3, Week 2
1
Where We Are
• Have selected ‘concept’ to learn
• Have chosen features to rep examples
• Have created at least 100 labeled examples
• Next: learn a ‘model’ that can predict output
for NEW examples
9/15/15
CS 540 - Fall 2015 (© Jude Shavlik), Lecture 3,
Week 2
Lecture 1, Slide 2
Learning from Labeled Examples
Positive Examples
Negative Examples
Category of this example?
Concept
Solid Red Circle in a (Regular?) Polygon
What about?
Figures on left side of page
Figures drawn before 5pm 2/2/89 <etc>
9/15/15
CS 540 - Fall 2015 (© Jude Shavlik), Lecture 3, Week 2
3
Concept Learning
Learning systems differ in how they represent concepts
Neural
Net
Backpropagation
Decision
Tree
ID3, C4.5, CART
Training
Examples
AQ, FOIL, Aleph
.
.
.
SVMs
9/15/15
CS 540 - Fall 2015 (© Jude Shavlik), Lecture 3, Week 2
ΦXY
ΦZ
Rules
Weighted Sum
If 5x1 + 9x2 – 3x3 > 12
Then +
4
Recall: Feature Space
If examples are described in terms of values
of features, they can be plotted as points in an
N-dimensional space
Size
Big
?
Gray
Color
2500
Weight
A “concept” is then a (possibly disjoint) volume in this space
9/15/15
CS 540 - Fall 2015 (© Jude Shavlik), Lecture 3, Week 2
5
Supervised Learning and Venn Diagrams
-
+
+ -
-
+
Venn Diagram
-
+
+
+
- + A
- Feature Space
-
-
-
+
-
+ +
+
+ + +
+
-
+
+
-
+
+
B
Concept = A or B (ie, a disjunctive concept)
Examples = labeled points in feature space
Concept = a label for regions of feat. space
9/15/15
CS 540 - Fall 2015 (© Jude Shavlik), Lecture 3, Week 2
6
Brief Introduction to Logic
• Conjunctive Concept
Instances
Color(?obj1, red)

“and”
Size(?obj1, large)
• Disjunctive Concept
Color(?obj2, blue)

“or”
Size(?obj2, small)
• More formally a “concept” is of the form
 x  y  z F(x, y, z)  Member(x, Class1)
9/15/15
CS 540 - Fall 2015 (© Jude Shavlik), Lecture 3, Week 2
7
Logical Symbols







9/15/15
and
or
not
implies
equivalent
for all
there exists
CS 540 - Fall 2015 (© Jude Shavlik), Lecture 3, Week 2
8
Induction vs. Deduction
Deduction
– compute what logically follows
– if we know P(Mary) is true and
x P(x)  Q(x), we can deduce Q(Mary)
Induction
– if we observe P(1), P(2), …, P(100)
we can induce x P(x)
– might be wrong
Which does supervised ML do?
9/15/15
CS 540 - Fall 2015 (© Jude Shavlik), Lecture 3, Week 2
9
Nearest-Neighbor Algorithms
(aka exemplar models, instance-based learning,
case-based learning) – Section 18.1.1 of textbook
• Learning ≈ memorize training examples
• Problem solving = find most similar example in
memory; output its category
Venn
-
“Voronoi
Diagrams”
(all points closest
to labeled example
in center)
+
-
-
+
+
+
…
-
-
-
+
+
9/15/15
+
+
?
+
CS 540 - Fall 2015 (© Jude Shavlik), Lecture 3, Week 2
-
+
10
Nearest Neighbors:
Basic Algorithm
• Find the K nearest neighbors to
test-set example
• Or find all ex’s within radius R
• Combine their ‘votes’
–
–
–
–
9/15/15
Most common category
Average value (real-valued prediction)
Can also weight votes by distance
Lots of variations on basic theme
CS 540 - Fall 2015 (© Jude Shavlik), Lecture 3, Week 2
+
- ?
+
-
11
Simple Example: 1-NN
(1-NN ≡ one nearest neighbor)
Training Set
1. a=0, b=0, c=1
2. a=0, b=0, c=0
3. a=1, b=1, c=1
Test Example
a=0, b=1, c=0
+
?
“Hamming Distance” (# of different bits)
Ex 1 = 2
So output Ex 2 = 1
Ex 3 = 2
9/15/15
CS 540 - Fall 2015 (© Jude Shavlik), Lecture 3, Week 2
12
Some Common Jargon
Classification
Learning a discrete valued function
Regression
Learning a real valued function
Discrete/Real
Outputs
(inputs can
be real valued
in both cases)
k-NN easily extended to regression tasks
(and to multi-category classification) – HOW?
9/15/15
CS 540 - Fall 2015 (© Jude Shavlik), Lecture 3, Week 2
13
Sample Experimental Results
(see UC-Irvine archive for more)
Testbed
1-NN
Testset Correctness
D-Trees
Neural Nets
Wisconsin
Cancer
98%
95%
96%
Heart Disease
78%
76%
?
Tumor
37%
38%
?
Appendicitis
83%
85%
86%
Simple algorithm works quite well!
9/15/15
CS 540 - Fall 2015 (© Jude Shavlik), Lecture 3, Week 2
Why so low?
14
Doing Well by
doing Poorly
• You say: “Bad news, my testset accuracy
is only 1%” (on a two-category task)
• I say: “That is great news!”
• Why?
• If you NEGATE your predictions,
you’ll have 99% accuracy!
9/15/15
CS 540 - Fall 2015 (© Jude Shavlik), Lecture 3, Week 2
15
Doing Poorly by
Doing Well
• You say: “Good news, my testset accuracy
is 95%” (on a two-category task)
• I say: “That is bad news!”
• Why might that be?
• Because (let’s assume) the most common
output value occurs 99% of the time!
9/15/15
CS 540 - Fall 2015 (© Jude Shavlik), Lecture 3, Week 2
16
Parameter Tuning (First Visit)
Algo: Collect K nearest neighbors, combine their outputs
What should K be?
– It is problem (ie, testbed) dependent
– Can use tuning sets to select
good setting for K
Shouldn’t really
“connect the dots”
(Why?)
Tuning Set
Error Rate
1
9/15/15
2
3
4
CS 540 - Fall 2015 (© Jude Shavlik), Lecture 3, Week 2
5
K
17
Why Not Use the TEST Set
to Select Good Parameters?
A 2002 paper in Nature (a major, major
journal) needed to be corrected due to
“training on the testing set”
Original report : 95% accuracy (5% error rate)
Corrected report (which still is buggy):
73% accuracy (27% error rate)
Error rate increased over 400%!!!
This is, unfortunately, a very common error
9/15/15
CS 540 - Fall 2015 (© Jude Shavlik), Lecture 3, Week 2
18
Some
ML “Commandments”
• Let the data decide
– ‘Internalize’ (ie, tune) parameters
• Scaling up by dummying down
– Don’t ignore simple algo’s, such as
• Always guessing most common category in the training set
• Find best SINGLE feature
– Clever ideas do not imply better results
• Generalize don’t memorize
– Accuracy on held-aside data is our focus
• Never train on the test examples!
– Commonly violated, alas
9/15/15
CS 540 - Fall 2015 (© Jude Shavlik), Lecture 3, Week 2
19