Introduction to Predictive Learning

LECTURE SET 4 Statistical Learning Theory

OUTLINE of Set 4

•

Objectives and Overview

• Inductive Learning Problem Setting • Keep-It-Direct Principle • Analysis of ERM • VC-dimension • Generalization Bounds • Structural Risk Minimization (SRM) • Summary and Discussion 2

Objectives

Problems with philosophical approaches - lack quantitative description/ characterization of ideas; - no real predictive power (as in Natural Sciences) - no agreement on basic definitions/ concepts (as in Natural Sciences)

Goal:

to introduce Predictive Learning as a scientific discipline 3

• • • • •

Characteristics of Scientific Theory

Problem setting Solution approach Math proofs (technical analysis) Constructive methods Applications

Note:

Problem Setting and Solution Approach are independent (of each other) 4

History and Overview

• SLT aka

VC-theory

(Vapnik-Chervonenkis) • Theory for estimating dependencies from finite samples (

predictive learning setting

) • Based on the

risk minimization

approach • All main results originally developed in 1970’s for classification (pattern recognition) –

why?

but remained largely unknown • Recent renewed interest due to practical success of

Support Vector Machines (SVM)

5

History and Overview(cont’d)

• • • •

MAIN CONCEPTUAL CONTRIBUTIONS Distinction

between problem setting, inductive principle and learning algorithms

Direct approach

principle ) to estimation with finite data ( KID

Math analysis of ERM

(standard inductive setting)

Two factors responsible for generalization:

empirical risk (fitting error) complexity(capacity) of approximating functions 6

Importance of VC-theory

• •

Math results addressing the main question

: - under what general conditions the ERM approach leads to (good) generalization?

New approach to induction:

Predictive vs generative modeling (in classical statistics) •

Connection to philosophy of science

VC-theory developed for binary classification (pattern recognition) ~ the simplest generalization problem natural sciences: from observations to scientific law  VC-theoretical results can be interpreted using general philosophical principles of induction, and vice versa.

7

Inductive Learning Setting

• The learning machine observes samples (

x

,

y

), and returns an estimated response

y



f

(

x

,

w

) • Two modes of inference: identification vs imitation • Risk 

Loss(y, f(

x

,w)) dP(

x

,y)



min

8

The Problem of Inductive Learning

•

Given:

finite training samples

Z

={(

x

i , y i ),i=1,2,…n} choose from a given set of functions f(x, w) one that

approximates best

the the true output. (in the sense of risk minimization)

Concepts and Terminology

• approximating functions f(x, w) • (non-negative) loss function

L

( f(x, w),y) • expected risk functional

R

(

Z

,

w

)

Goal:

find the function

f

(

x

,

w

o )

minimizing R

(

Z

,

w

) when the joint distribution

P

(

x

,y) is

unknown.

9

Empirical Risk Minimization

• ERM principle in model-based learning – Model parameterization: f(x, w) – Loss function:

L

( f(x, w),y) – Estimate risk from data:

R emp

(

w

)  1 – Choose

w

* that minimizes

R emp n i n

  1

L

(

f

(

x

i

,

w

),

y i

) • Statistical Learning Theory developed from the theoretical analysis of ERM principle under finite sample settings 10

Probabilistic Modeling vs ERM 11

Probabilistic Modeling vs ERM: Example Known class distribution  optimal decision boundary 10 8 6 4 2 0 -2 -4 -6 -2 0 2 4 x1 6 8 10 12

Probabilistic Approach Estimate parameters of Gaussian class distributions , and plug them into quadratic decision boundary 0 -2 4 2 10 8 6 -4 -6 -2 0 2 4 x1 6 8 10 13

ERM Approach Quadratic and linear decision boundary estimated via minimization of squared loss -2 -4 -6 -2 4 2 0 10 8 6 0 2 4 x1 6 8 10 14

Estimation of multivariate functions

• Is it possible to estimate a function from finite data? •

Simplified problem:

estimation of unknown continuous function from noise-free samples • Many results from function approximation theory : – To estimate accurately a

d

-dimensional function one needs O(n^^d) data points – For example, if 3 points are needed to estimate 2-nd order polynomial for d=1, then 3^^10 points are needed to estimate 2-nd order polynomial in 10-dimensional space.

– Similar results in signal processing • Never enough data points to estimate multivariate functions in most practical applications (image recognition, genomics etc.) • For multivariate function estimation, the number of free parameters increases exponentially with problem dimensionality (the Curse of Dimensionality) 15

Properties of high-dimensional data • • •  Sparse data looks like a

porcupine

: the volume of a unit sphere inscribed in a d dimensional cube gets

smaller

even as the volume of d-cube gets

exponentially larger

!

A point is

closer to an edge

than to another point Pairwise distances between points are

the same

Intuition behind kernel (local) methods no longer holds.

• How generalization is possible, in spite of the curse of dimensionality ?

16

OUTLINE of Set 4

• Objectives and Overview • Inductive Learning Problem Setting •

Keep-It-Direct Principle

• Analysis of ERM • VC-dimension • Generalization Bounds • Structural Risk Minimization (SRM) • Summary and Discussion 17

• • • •

Keep-It-Direct Principle

The goal of learning is generalization rather than estimation of true function (system identification) 

Loss(y, f(

x

,w)) dP(

x

,y)



min

Keep-It-Direct Principle

(Vapnik, 1995)

Do not solve an estimation problem of interest by solving a more general (harder) problem as an intermediate step

Good predictive model reflects unknown distribution

P

(

x

,y) some properties of Since model estimation with finite data is ill-posed, one should never try to solve a more general problem than required by given application  Importance of formalizing application requirements as a learning problem.

18

•

Learning vs System Identification

Consider regression problem where unknown target function

y g

 (

x

)

g

( 

x

E

) ( 

y

 /

x

) • Goal 1 : Prediction

R

(

w

)   (

y



f

(

x

,

w

)) 2

dP

(

x

,

y

)  min • Goal 2 : Function Approximation (system identification) or

R

(

w

)

f

 (

x

, 

w

) (

f

 (

x

,

E

(

w

)

y

/ 

x

)

g

(

x

 )) 2

d

min

x

 min • • Admissible models : algebraic polynomials Purpose of comparison : contrast goals (1) and (2)

NOTE:

most applications assume Goal 2, i.e.

Noisy Data ~ true signal + noise 19

•

Empirical Comparison

Target function: sine-squared

g

(

x

)  sin 2 ( 2 

x

)

x

 [ 0 , 1 ] • • 0.4

0.3

0.2

0.1

0 0 1 0.9

0.8

0.7

0.6

0.5

0.2

0.4

0.6

0.8

1 Input distribution : non-uniform Gaussian pdf Additive gaussian noise with st. deviation = 0.1

20

• • • •

Empirical Comparison (cont’d)

Model selection: use separate data sets

training

: for parameter estimation -

validation

: for selecting polynomial degree -

test

: for estimating prediction risk (MSE) Validation set generated differently to contrast (1)&(2) Predictive Learning (1) ~ Gaussian Funct. Approximation (2) ~ uniform fixed sampling Training + test data ~ Gaussian Training set size : 30 Validation set size : 30 21

• Regression estimates (2 typical realizations of data): Dotted line ~ estimate obtained using predictive learning Dashed line ~ estimate via function approximation setting 22

• • •

Conclusion

The goal of prediction (1) is different (less demanding) than the goal of estimating the true target function (2) everywhere in the input space.

The curse of dimensionality applies to system identification setting (2), but may not hold under predictive setting (1).

Both settings coincide if the input distribution is uniform (i.e., in signal and image denoising applications) 23

Philosophical Interpretation of KID • • •

Interpretation of predictive models

Realism

~ objective truth (hidden in Nature)

Instrumentalism

~ creation of human mind (imposed on the data) – favored by KID

Objective Evaluation

prediction risk reflecting application needs)  Natural Science still possible (via • •

Methodological implications

Importance of good learning formulations (asking the ‘right question’) Accounts for 80% of success in applications 24

OUTLINE of Set 4

• Objectives and Overview • Inductive Learning Problem Setting • • Keep-It-Direct Principle

Analysis of ERM

• VC-dimension • Generalization Bounds • Structural Risk Minimization (SRM) • Summary and Discussion 25

VC-theory has 4 parts:

1. Analysis of consistency/convergence of ERM

R emp

 1

n i n

  1

L y f i

x ,

i

 ))  min 2. Generalization bounds 3. Inductive principles (for finite samples) 4. Constructive methods (learning algorithms) for implementing (3)

NOTE:

(1)  (2)  (3)  (4) 26

Consistency/Convergence of ERM

• • Empirical Risk known but Expected Risk unknown

Asymptotic consistency requirement

: under what (general) conditions models providing

min Empirical Risk

will also provide

min Prediction Risk

, when the number of samples grows large?

•

Why asymptotic analysis

is needed?

- helps to develop useful concepts - necessary and sufficient conditions ensure that VC-theory is general and can not be improved 27

Consistency of ERM

does not imply consistency

 

R

  of ERM • Models estimated via ERM (w*) are

always biased estimates

of the functions minimizing true risk:

R emp

   

n n

28

• •

Conditions for Consistency of ERM

Main insight

: consistency is not possible without restricting the set of possible models

Example:

1-nearest neighbor classification method.

- is it consistent ?

• Consider binary decision functions (classification) • How to measure their flexibility, or ability to ‘explain’ the training data (for binary classification)?

• This complexity index for indicator functions: - is independent of unknown data distribution ; - measures the capacity of a set of possible models , rather than characteristics of the ‘true model’ 29

OUTLINE of Set 4

• Objectives and Overview • Inductive Learning Problem Setting • Keep-It-Direct Principle • Analysis of ERM •

VC-dimension

• Generalization Bounds • Structural Risk Minimization (SRM) • Summary and Discussion 30

• SHATTERING

Linear indicator functions

: can split 3 data points in 2D in all

2^^3 =

8 possible binary partitions • If a set of n samples can be separated by a set of functions in all

2^^n

said to be shattered possible ways, this sample is (by the set of functions) • Shattering ~ a set of models can explain a given sample of size

n

(for all possible labelings) 31

• VC DIMENSION

Definition

: A set of functions has VC-dimension

h

is there exist

h

samples that can be shattered by this set of functions, but there are no

h+1

samples that can be shattered •  VC-dimension

h

=3 (

h=d+1

for linear functions ) • VC-dim. is a positive integer (combinatorial index) • What is VC-dim. of 1-nearest neighbor classifier ?

32

VC-dimension and Consistency of ERM • •

VC-dimension is infinite

if a sample of size n can be split in all 2^^n possible ways (in this case, no valid generalization is possible)

Finite VC-dimension

sufficient conditions gives for: necessary and (1) consistency of ERM-based learning (2) fast rate of convergence (these conditions are distribution-independent) • Interpretation of the VC-dimension via

falsifiability:

functions with small VC-dim can be easily falsified 33

VC-dimension and Falsifiability

• A set of functions has VC-dimension

h

if (a) It can explain (shatter) a set of

h

samples ~ there exists

h

samples that cannot falsify it

and

(b) It can not shatter

h+1

samples ~ any

h+1

samples falsify this set •

Finiteness of VC-dim

is necessary and sufficient condition for generalization (for

any

learning method based on ERM) 34

Recall Occam’s Razor:

• • Main problem in predictive learning - Complexity control (model selection) - How to measure complexity ?

Interpretation of Occam’s razor

(in Statistics): Entities ~ model parameters Complexity ~ degrees-of-freedom  Necessity ~ explaining (fitting) available data Model complexity = number of parameters (DoF) • Consistent with classical statistical view: learning = function approx. / density estimation 35

Philosophical Principle of VC-falsifiability • Occam’s Razor : Select the model that explains available data

and

has the smallest number of free parameters (entities) • VC theory : Select the model that explains available data

and

has low VC-dimension (i.e. can be easily falsified )  New principle of VC-falsifiability 36

Calculating the VC-dimension • How to estimate the VC-dimension (for a given set of functions)?

• Apply definition (via shattering) to derive analytic estimates – works for ‘simple’ sets of functions • Generally, such analytic estimates are

not possible

for complex nonlinear parameterizations (i.e., for practical machine learning and statistical methods) 37

• Example: VC-dimension of spherical indicator functions . Consider spherical decision surfaces in a

d

-dimensional

x

space, parameterized by center

f



r

 

I

 (

c

 ) 2 and radius

r



r

2  parameters: In a 2-dim space (d=2) there exists 3 points that can be shattered, but 4 points cannot be shattered  h=3 38

• Example: functions VC-dimension of a linear combination of (i.e. polynomials, Fourier expansion etc.) fixed basis Assuming that basis functions are linearly independent, the VC-dim equals the number of basis functions (free parameters).

• Example: single parameter but infinite VC-dimension

f



x

,

w

 

I



sin

wx

 0  39

• Example: Wide linear decision boundaries

D

   (

w



x

) 

b

Consider linear functions such that the distance btwn D(x) and the closest data sample is larger than given value  Then VC-dimension depends on the width parameter, rather than

d

(as in linear models):

h

 min   

R

2  2 ,

d

    1 40

Linear combination of fixed basis functions

f



x, w

 

I

 

i m

  1

w g i i



w

0  0   is equivalent to linear functions in m-dimensional space  VC-dimension = m + 1 (this assumes linear independence of basis functions) • In general, analytic estimation of VC-dimension is hard • VC-dimension can be - equal to DoF - larger than DoF - smaller than DoF 41

VC-dimension vs number of parameters • VC-dimension can be equal parameters) to DoF (number of Example: linear estimators • VC-dimension can be smaller than DoF Example: penalized estimators • VC-dimension can be larger than DoF Example: feature selection

sin (wx)

42

VC-dimension for Regression Problems • VC-dimension was defined for indicator functions • Can be extended to real-valued functions, i.e.

third-order polynomial for univariate regression:

f

(

x

,

w

,

b

) 

w

3

x

3  linear parameterization 

w

2

x

2 

w

1

x

VC-dim = 4 

b

• Qualitatively, the VC-dimension ~ the ability to fit (or explain) finite training data for regression.

This can be also related to falsifiability 43

Example: what is VC-dim of kNN Regression? • • Ten training samples from

y



x

2  0 .

1

x



N

( 0 ,  2 ),

where

 2  0 .

25 Using k-nn regression with k=1 and k=4: 44

OUTLINE of Set 4

• Objectives and Overview • Inductive Learning Problem Setting • Keep-It-Direct Principle • Analysis of ERM • • VC-dimension

Generalization Bounds

• Structural Risk Minimization (SRM) • Summary and Discussion 45

Recall consistency of ERM Two Types of VC-bounds:

emp

 

R

  (2) How close is the empirical risk to the minimal possible risk ?

46

• •

Generalization Bounds

Bounds for learning machines

(implementing ERM) evaluate the difference btwn (unknown) risk and known empirical risk, as a function of sample size

n

and general properties of admissible models (their VC-dimension)

Classification:

the following bound holds with probability 

all approximating functions R

(  ) 

R emp

where  (  )   

R emp

(  ),

n

/

h

,  ln  /

n

 is called the confidence interval •

Regression:

1   the following bound holds with probability of for all approximating functions

R

(  ) 

R emp

(  ) / where  1 

c

       

n h

,



ln



n

 

a

1

h

 

ln

a

2

h n

 1  

n

ln

 

/

4  47

Practical VC Bound for regression

•

Practical regression bound

  theoretical constants:

R

(

h

) 

R emp

(

h

)    1 

h n



h n

ln

h n

 ln 2

n n

     1 can be used for model selection (examples given later) • Compare to analytic bounds (SC, FPE) in Lecture Set 2 • Analysis (of denominator) shows that

h < 0.8 n for any estimator

In practice:

h < 0.5 n for any estimator

48

VC Regression Bound for model selection • VC-bound can be used for

analytic model selection

(if the VC-dimension is known) Example: polynomial regression for estimating Sine_Squared target function from 25 noisy samples 1.5

Optimal model found: 6-th degree polynomial (no resampling needed) 1 0.5

0 -0.5

0 0.2

0.4

x

0.6

0.8

1 49

Modeling pure noise with x in [0,1] via poly regression sample size

n

=30, noise   1

10 10 10 0 10 -10 fpe gcv vc cv 30 20 10 0 fpe gcv vc cv

Comparison of different model selection methods: - prediction risk (MSE) - selected DoF (~

h

) 50

OUTLINE of Set 4

• Objectives and Overview • Inductive Learning Problem Setting • Keep-It-Direct Principle • Analysis of ERM • VC-dimension • Generalization Bounds •

Structural Risk Minimization (SRM)

• Summary and Discussion 51

Structural Risk Minimization

•

Analysis of generalization bounds

R

(  ) 

R emp

(  )   

R emp

(  ),

n

/

h

,  ln  /

n

 suggests that when

n/h



R

(  ) is large , the term is small ~

R emp

(  ) This leads to parametric modeling approach (ERM) • When

n/h

is not large (say, less than 20), both terms right-hand side of VC- bound need to be minimized  make the VC-dimension a controlling variable in the • SRM = formal mechanism for controlling model complexity Set of admissible models has a nested structure

S

1   ...



S k

 ...

such that

h

1 

h

2 ...



h k

...



S

2 structure formally defines complexity ordering 52

Structural Risk Minimization

• An upper bound on the true risk and the empirical risk, as a function of VC-dimension

h

(for fixed sample size

n

) 53

SRM vs ERM modeling

54

SRM Approach

• Use

VC-dimension

minimizing as a controlling parameter for VC bound: ( ( ) ) ( (  ) )   / /   •

Two general strategies for implementing SRM

:  

R emp

(  ) (most statistical and neural network methods)  (Support Vector Machines)    55

Common SRM structures

•

Dictionary structure

A set of algebraic polynomials is a structure since

f

1 

f

2

f



....



m



f

k x

,

w

 

....



b



i m

  0

w i x i

More generally where

g



x

,

v

i



f m



x

,

w

,

V

 

b



m

 

w i g



x

,

v

i



i

0 is a set of basis functions ( dictionary ).

The number of terms (basis functions)

m

specifies an element of a structure.

For fixed basis fcts, VC-dim ~ number of parameters

w i

56

•

Common SRM structures

Feature selection

(aka subset selection) Consider

sparse polynomials

of degree

m

: for

m

=1:

f

1 (

x

,

w

,

b

,

k

1 ) 

b



wx k

1 for

m

=2:

f

2 (

x

,

w

,

b

,

k

1 ,

k

2 ) 

b



w

1

x k

1 

w

2

x k

2 etc.

• Each monomial is a feature. The goal is to select a set of

m

features providing min. empirical risk (MSE) This is a structure since

f 1



f 2



....



f m



...

.

• More generally , where

m

f

m



x

,

w

,

V

 

m



w i

g

 

i

 0 basis fcts are selected from a (large) set of M fcts

Note:

nonlinear optimization, VC-dimension is unknown 57

•

Common SRM structures

Penalization

Consider algebraic polynomial of fixed degree

f



x

,

w

 

w i x i

where

w

2 

c k c

1 

i

10   0

c

2 

c

3 ...

For each (positive) value

c

an element of a structure this set of functions specifies

S k

 

f



x

,

w



,

w

2 

c k

 Minimization of empirical risk (MSE) on each element

k

of a structure is a constrained minimization problem This optimization problem can be equivalently stated as minimization of the penalized empirical risk functional :

R pen

  , 

k

 

R emp

 

k

w

2 where the choice of 

k

~

c k

Note:

VC-dimension is unknown 58

Example: SRM structures for regression • • •

Regression data set

x-values~ uniformly sampled in [0,1] y-values ~ target fct

y

 0 .

8 sin( 2 

x

)  0 .

2

x

2  0 .

5 additive Gaussian noise with st. dev 0.05

Experimental set-up

training set ~ 40 samples

x

validation set ~ 40 samples (for model selection)

SRM structures

defined on algebraic polynomials - dictionary (polynomial degrees 1 to 10) - penalization (fixed degree-10 polynomial) - sparse polynomials (degree 1 to 5) 59

Estimated models

using different SRM structures: - dictionary  0 .

4078  6 .

4198

x

 68 .

2162

x

2  163 .

7679

x

3  158 .

3952

x

4  55 .

9565

x

5 - penalization

lambda=1.013e-005

- sparse polynomial  0 .

6186  22 .

7337

x

2  41 .

1772

x

3  19 .

2736

x

4 Visual results : target fct~ red line, feature selection~ black solid, dictionary ~ green, penalization ~ yellow line 1 0.5

-0.5

-1 0 -1.5

0 0.2

0.4

x

0.6

0.8

1 60

SRM Summary

• •

SRM structure

~ complexity ordering on a set of admissible models (approximating functions) •

Many different structures

on the same set of approximating functions (possible models) • How to choose the ‘best’ structure?

- depends on application data - VC theory cannot provide answer

SRM = mechanism for complexity control

- selecting optimal complexity for a given data set - new measure of complexity: VC-dimension - model selection using analytic VC-bounds 61

OUTLINE of Set 4

• Objectives and Overview • Inductive Learning Problem Setting • Keep-It-Direct Principle • Analysis of ERM • VC-dimension • Generalization Bounds • • Structural Risk Minimization (SRM)

Summary and Discussion

62

• • • • •

Summary and Discussion: VC-theory

-

Methodology

learning problem setting concepts structure) (KID principle) (risk minimization, VC-dimension,

Interpretation/ evaluation of existing methods Model selection

using VC-bounds

New types of inference

(TBD later) -

What theory can not do

: provide formalization select ‘good’ structure (for a given application) always a gap between theory and applications 63