Transcript [ppt]
Machine Learning Reading: Chapter 18 Text Classification Is texti a finance new article? Positive Negative 2 20 attributes Investors Dow Jones Industrial Average Percent Gain Trading Broader stock Indicators Standard Rolling Nasdaq Early Rest More first Same The 2 2 2 1 3 5 6 8 5 5 6 2 1 3 10 12 13 11 12 30 3 20 attributes Men’s Basketball Championship UConn Huskies Georgia Tech Women Playing Crown Titles Games Rebounds All-America early rolling Celebrates Rest More First The same 4 Example stock rolling the class 1 0 3 40 other 2 6 8 35 finance 3 7 7 25 other 4 5 7 14 other 5 8 2 20 finance 6 9 4 25 finance 7 5 6 20 finance 8 0 2 35 other 9 0 11 25 finance 10 0 15 28 other Constructing the Decision Tree Goal: Find the smallest decision tree consistent with the examples Find the attribute that best splits examples Form tree with root = best attribute For each value vi (or range) of best attribute Selects those examples with best=vi Construct subtreei by recursively calling decision tree with subset of examples, all attributes except best Add a branch to tree with label=vi and subtree=subtreei 6 Choosing the Best Attribute: Binary Classification Want a formal measure that returns a maximum value when attribute makes a perfect split and minimum when it makes no distinction Information theory (Shannon and Weaver 49) Entropy: a measure that characterizes the impurity of a collection of examples Information gain: the expected reduction in entropy caused by partitioning the xamples according to this attribute 7 Formula for Entropy n H(P(v1),…P(vn))=∑-P(vi)log2P(vi) i=1 where P(v) = probability of v Examples: Suppose we have a collection of 10 examples, 5 positive, 5 negative: H(1/2,1/2)=-1/2log21/2-1/2log21/2=1 bit Suppose we have a collection of 100 examples, 1 positive and 99 negative: H(1/100,99/100)=-.01log2.01-.99log2.99=.08 bits 8 Choosing the Best Attribute: Information Gain Information gain (from attribute test) = difference between the original information requirement and new requirement Gain(A)=H(p/p+n,n/p+n)-Remainder(A) H=entropy Highest when the set is equally divided between positive (p) and negative (n) examples (.5,.5) (value of 1) Lower as the set becomes more unbalanced (e.g., (.9, .1) ) 9 Information based on attributes P=n=10, so H(1/2,1/2)= 1 bit = Remainder (A) Text Classification Is texti a finance new article? Positive Negative 11 Example stock rolling the class 1 0 3 40 other 2 6 8 35 finance 3 7 7 25 other 4 5 7 14 other 5 8 2 20 finance 6 9 4 25 finance 7 5 6 20 finance 8 0 2 35 other 9 0 11 25 finance 10 0 15 28 other stock <5 5-10 rolling 10 <5 1,5,6,8 1,8,9,10 2,3,4,5,6,7 10 5-10 9,10 2,3,4,7 Algorithm as specified so far is designed for binary classification, attributes with discrete values Attributes: Outlook: sunny, overcast, rain Temperature: hot, mild, cool Humidity: normal, high Wind: weak, strong Classification PlayTennis?: Yes, No 14 Day Outlook Temperatu re Humidity Wind PlayTennis D1 Sunny Hot High Weak No D2 Sunny Hot High Strong No D3 Overcast Hot High Weak Yes D4 Rain Mild High Weak Yes D5 Rain Cool Normal Weak Yes D6 Rain Cool Normal Strong No D7 Overcast Cool Normal Strong Yes D8 Sunny Mild High Weak No D9 Sunny Cool Normal Weak Yes D10 Rain Mild Normal Weak Yes D11 Sunny Mild Normal Strong Yes D12 Overcast Mild High Strong Yes D13 Overcast Hot Normal Weak Yes D14 Rain Mild High Strong No Humidity Wind E=.940 (9/14 yes) E=.94 High E=.985 Normal E=.592 Gain(humidity)=.940-(7/14).985-(7/14).592 Outlook Strong E=.811 E=.1.0 Gain(wind)=.940-(6/14).811-(8/14)1.0 Temperature E=.940 E=.940 Overcast Weak Sunny Cool Rain Gain(outlook)=.940-(4/14)0-(5/14).79(5/14).79 Gain(S,Outlook)=.246, Gain(S,Humidity)=.151, Gain(S,Wind)=.048, Gain(S,Temperature)=.029 Outlook is selected because it has highest gain Mild Hot Day Outlook Temperatu re Humidity Wind PlayTennis D1 Sunny Hot High Weak No D2 Sunny Hot High Strong No D3 Overcast Hot High Weak Yes D4 Rain Mild High Weak Yes D5 Rain Cool Normal Weak Yes D6 Rain Cool Normal Strong No D7 Overcast Cool Normal Strong Yes D8 Sunny Mild High Weak No D9 Sunny Cool Normal Weak Yes D10 Rain Mild Normal Weak Yes D11 Sunny Mild Normal Strong Yes D12 Overcast Mild High Strong Yes D13 Overcast Hot Normal Weak Yes D14 Rain Mild High Strong No Day Outlook Temperatu re Humidity Wind PlayTennis D1 Sunny Hot High Weak No D2 Sunny Hot High Strong No D3 Overcast Hot High Weak Yes D4 Rain Mild High Weak Yes D5 Rain Cool Normal Weak Yes D6 Rain Cool Normal Strong No D7 Overcast Cool Normal Strong Yes D8 Sunny Mild High Weak No D9 Sunny Cool Normal Weak Yes D10 Rain Mild Normal Weak Yes D11 Sunny Mild Normal Strong Yes D12 Overcast Mild High Strong Yes D13 Overcast Hot Normal Weak Yes D14 Rain Mild High Strong No Extending the algorithm for continuous valued attributes Dynamically define new discrete-valued attributes that partition the continuous attribute into a discrete set of intervals For continuous A, create Ac that is true if A<c, false otherwise How to select the best value for threshold c? Sort examples by continuous attribute Identify adjacent examples that differ in target classification Generate a set of candidate thresholds midway between corresponding values of A Choose threshold c that maximizes information gain 19 Example: temperature as continuous value Temp 40 48 60 72 80 90 Play tennis? No No Yes Yes Yes No Two candidate thresholds: (48+60)/2 (80+90)/2 Information gain greater for Temperature>54 than for Temperature>85 20 Other cases What if class is discrete valued, not binary? What if an attribute has many values (e.g., 1 per instance)? 21 Training vs. Testing A learning algorithm is good if it uses its learned hypothesis to make accurate predictions on unseen data Collect a large set of examples (with classifications) Divide into two disjoint sets: the training set and the test set Apply the learning algorithm to the training set, generating hypothesis h Measure the percentage of examples in the test set that are correctly classified by h Repeat for different sizes of training sets and different randomly selected training sets of each size. 22 23 Overfitting Learning algorithms may use irrelevant attributes to make decisions For news, day published and newspaper When else can overfitting occur? Solution #1: Decision tree pruning Prune away attributes with low information gain Use statistical significance to test whether gain is meaningful 24 K-fold Cross Validation Solution #2: To reduce overfitting Run k experiments Use a different 1/k of data for testing each time Average the results 5-fold, 10-fold, leave-one-out 25 Cross-Validation Labeled data (1566) Split into 10 folds 9 folds (approx. 1409) Train 1 fold (approx. 157) Model Lather, rinse, repeat (10 times) Evaluate Report average 26 Example 27 Ensemble Learning Learn from a collection of hypotheses Majority voting Enlarges the hypothesis space 28 Boosting Uses a weighted training set Each example has an associated weight wj0 Higher weighted examples have higher importance Initially, wj=1 for all examples Next round: increase weights of misclassified examples, decrease other weights From the new weighted set, generate hypothesis h2 Continue until M hypotheses generated Final ensemble hypothesis = weighted-majority combination of all M hypotheses Weight each hypothesis according to how well it did on training data 29 AdaBoost If input learning algorithm is a weak learning algorithm L always returns a hypothesis with weighted error on training slightly better than random Returns hypothesis that classifies training data perfectly for large enough M Boosts the accuracy of the original learning algorithm on training data 30