Transcript Analysis of Variance in Matrix form

Analysis of Variance in Matrix form
• The ANOVA table sums of squares, SSTO, SSR and SSE can all be
expressed in matrix form as follows….
week 9
1
Multiple Regression
• A multiple regression model is a model that has more than one
explanatory variable in it.
• Some of the reasons to use multiple regression models are:
 Often multiple X’s arise naturally from a study.
 We want to control for some X’s
 Want to fit a polynomial
 Compare regression lines for two groups
week 9
2
Multiple Linear Regression Model
• In a multiple linear regression model there are k predictor variables.
• The model is
Yi   0  1 X i 1   2 X i 2     k X i k   i ,
i  1, ...,n
• This model is linear in the β’s. The variables may be non-linear, e.g.,
log(X1), X1*X2 etc.
• We need to estimate k +1 β’s and σ2.
• There are k+2 parameters in this model and so we need at least that
many observations to be able to estimate them.
week 9
3
Multiple Regression Model in Matrix Form
• In matrix notation the multiple regression model is: Y=Xβ + ε
where
Y1 
 1 
 0 
1 X 11  X 1k 
Y 
 
 
1 X

X
2
2
21
2
k
1

Y  ,    ,    , X
 
 
 


 
 
 
 


1
X
X
Y


n1
nk 

 n
 n
 
• Note, Y and ε are n  1 vectors, β is a k  1  1 vector and X is a
n  k  1 matrix.
• The Gauss-Markov assumptions are: E(ε) = 0, Var(ε) = σ2I.
• These result in E(Y) = 0, Var(Y) = σ2I.
1
• The Least-Square estimate of β is b  X' X X' Y.
week 9
4
Estimate of σ2
• The estimate of σ2 is:
s 
2
2
e
i
df
e' e

n  k 1
• It has n-k-1 degrees of freedom because…
• Claim: s2 is unbiased estimator of σ2.
• Proof:
week 9
5
General Comments about Multiple Regression
• The regression equation gives the mean response for each
combination of explanatory variables.
• The regression equation will not be useful if it is very complicated
or a function of large number of explanatory variables.
• We generally want “parsimonious” model, that is, a model that is as
simple as possible to adequately describe the response variable.
• It is unwise to think that there is some exact, discoverable equation.
• Many possible models are available.
• One or two models may adequately approximate the mean of the
response variable.
week 9
6
Example – House Prices in Chicago
• Data of 26 house sales in Chicago were collected (clearly collected
some time ago). The variables in the data set are:
price - selling price in $1000's
bdr - number of bedrooms
flr - floor space in square feet
fp - number of fireplaces
rms - number of rooms
st - storm windows (1 if present, 0 absent)
lot - lot size (frontage) in feet
bth - number of bathrooms
gar - garage size (0=no garage, 1=one-car garage, etc.)
week 9
7
Interpreting Regression Coefficients
• b0 is the estimate of the mean value of Y when all the predictor
variables are 0; it may be meaningless in many cases.
• bj , j = 1,…, k is the estimated change in Y, on average, due to a unit
increase in Xj where all the other variables held fixed. Note, that it
may be impossible to hold all other variables fixed.
• Example, re the home price example above, for 100 extra square
feet (everything else held constant), the price goes up by $1760 on
average. For one more room (everything else held constant), the
price goes up by $3900 on average. For one more bedroom
(everything else held constant), the price goes down by $7700 on
average.
week 9
8
Residual Plots
• As in the simple linear regression model, we examine 3 types of
residual plots to assess the validity of the model assumptions.
• The plots are:
 Residual versus Xj , j = 1, …, k. We examine presence of
curvature and/or influential points.
 Residual versus predicted values. We use this plot to check for
non-constant variance and for any outliers in residuals.
 Normal quantile plot of residuals.
week 9
9
Inference for Regression Coefficients
• As in simple linear regression, we are interesting in testing H0: βj = 0
versus Ha: βj ≠ 0.
• The test statistics is
t stat 
bj
S .E b j 
It has a t-distribution with n-k-1 degrees of freedom.
• We can calculate the P-value from the t-table with n-k-1 df.
• This test gives an indication of whether or not the jth predictor
variable statistically significant contributes to the prediction of the
response variable over and above all the other predictor variables.
• Confidence interval for βj (assuming all the other predictor variables
are in the model) is given by: b j  t nk 1; / 2  S.Eb j 
• We can use Bonferroni correction for simultaneous CI for more than
one β.
week 9
10
ANOVA Table
• The ANOVA table in multiple regression model is given by…
week 9
11
Coefficient of Multiple Determination – R2
• As in simple linear regression model, R2 = SSR/SSTO.
• In multiple regression this is called the “coefficient of multiple
determination”; it is not the square of a correlation coefficient.
• In multiple regression, R2 always goes up when more predictor
variables are added to the model, regardless of whether the predictor
variables are useful in estimating Y.
week 9
12
Adjusted R2
• An attempt to make R2 more useful is to calculate Adjusted R2
(“Adj R-Sq” in SAS)
• Adjusted R2 is adjusted for the number of predictor variables in the
model.
• It can actually go down when more predictors are added.
• It can be used for choosing the best model.
• It is defined as
Adj R 2  1 
n  1MSE
SSTO
 1
n  1 SSE
.
n  k  1 SSTO
• Note that Adjusted R2 will increase only is MSE decrease.
week 9
13