Transcript Slides
Machine Learning Feature Selection Feature Selection for Pattern Recognition J.-S. Roger Jang ( 張智星 ) CSIE Dept., National Taiwan University ( 台灣大學 資訊工程系 ) http://mirlab.org/jang [email protected] Machine Learning Feature Selection Feature Selection: Goal & Benefits Feature selection • Also known as input selection Goal • To select a subset out of the original feature sets for better recognition rate Benefits • Improve recognition rate • Reduce computation load • Explain relationships between features and classes 2 Machine Learning Feature Selection Exhaustive Search Steps for direct exhaustive search 1. Use KNNC as the classifier, LOO for RR estimate 2. Generate all combinations of features and evaluate them one-by-one 3. Select the feature combination that has the best RR. Drawback • d features 2d 1 modelsfor evaluation • d = 10 1023 models for evaluation Time consuming! Advantage • 3 The optimal feature set can be identified. Machine Learning Feature Selection Exhaustive Search Direct exhaustive search x2 x3 x4 x5 x1, x2 3 inputs x1, x2, x3 x1, x2, x4 x1, x2, x5 x1, x3, x4 4 inputs x1, x2, x3, x4 x1, x2, x3, x5 x1, x2, x4, x5 x1, x4 x1, x5 x2, x3 .. . x1, x3 .. . 2 inputs .. . 4 x1 .. . 1 input Machine Learning Feature Selection Exhaustive Search Characteristics of exhaustive search for feature selection • • • 5 The process is time consuming, but the identified feature set is optimum. It’s possible to use classifiers other than KNNC. It’s possible to use performance indices other than LOO. Machine Learning Feature Selection Heuristic Search Heuristic search for input selection • • • • • • • 6 One-pass ranking Sequential forward selection Generalized sequential forward selection Sequential backward selection Generalized sequential backward selection ‘Add m, remove n’ selection Generalized ‘add m, remove n’ selection Machine Learning Feature Selection Sequential Forward Selection Steps for sequential forward selection 1. Use KNNC as the classifier, LOO for RR estimate 2. Select the first feature that has the best RR. 3. Select the next feature (among all unselected features) that, together with the selected features, gives the best RR. 4. Repeat the previous step until all features are selected. Advantage • If we have d features, we need to evaluate d(d+1)/2 models A lot more efficient. Drawback 7 • The selected features are not always optimal. Machine Learning Feature Selection Sequential Forward Selection Sequential forward selection (SFS) 1 input x2 x3 x4 x5 2 inputs x2, x1 3 inputs x2, x4, x1 x2, x4, x3 x2, x4, x5 4 inputs x2, x4, x3, x1 x2, x4, x3, x5 .. . 8 x1 x2, x3 x2, x4 x2, x5 Machine Learning Feature Selection Example: Iris Dataset Sequential forward selection 9 Exhaustive search Machine Learning Feature Selection Example: Wine Dataset SFS 3 selected features, LOO RR=93.8% 10 SFS with input normalization 6 selected features, LOO RR=97.8% If we use exhaustive search, we have 8 features with LOO RR=99.4% Machine Learning Feature Selection Use of Input Selection Common use of input selection • Increase the model complexity sequentially by adding more inputs • Select the model that has the best test RR Typical curve of error vs. model complexity • Determine the model structure with the least test error Error rate Test error 11 Optimal structure Training error Model complexity (# of selected inputs)