Transcript Econometrics I Professor William Greene Stern School of Business Department of Economics 7-1/53 Part 7: Estimating the Variance of b.

Econometrics I
Professor William Greene
Stern School of Business
Department of Economics
7-1/53
Part 7: Estimating the Variance of b
Econometrics I
Part 7 – Estimating
the Variance of b
7-2/53
Part 7: Estimating the Variance of b
Context
The true variance of b|X is 2(XX)-1 . We
consider how to use the sample data to estimate
this matrix. The ultimate objectives are to form
interval estimates for regression slopes and to
test hypotheses about them. Both require
estimates of the variability of the distribution.
We then examine a factor which affects how
"large" this variance is, multicollinearity.
7-3/53
Part 7: Estimating the Variance of b
Estimating 2
Using the residuals instead of the disturbances:
The natural estimator: ee/N as a sample
surrogate for /n
Imperfect observation of i, ei = i - ( - b)xi
Downward bias of ee/N. We obtain the result
E[ee|X] = (N-K)2
7-4/53
Part 7: Estimating the Variance of b
Expectation of ee
e
 y - Xb
 y  X ( X ' X ) 1 X ' y
1
 [I  X ( X ' X ) X ']y
 My  M( X  )  MX  M  M
e'e  (M'(M
 'M'M  'MM  'M
7-5/53
Part 7: Estimating the Variance of b
Method 1:
E[e'e|X ]  E'M |X 
E[ trace ('M |X ) ] scalar = its trace
E[ trace (M'|X ) ] permute in trace
[ trace E (M'|X ) ] linear operators
[ trace M E ('|X ) ] conditioned on X
[ trace M 2I ] model assumption
2 [trace M ] scalar multiplication and I matrix
2 trace [I - X( X'X)-1 X' ]
2 {trace [I] - trace[X( X'X)-1 X' ]}
2 {N - trace[( X'X)-1 X'X ]} permute in trace
2 {N - trace[I]}
2 {N - K}
Notice that E[ee|X] is not a function of X.
7-6/53
Part 7: Estimating the Variance of b
Estimating σ2
The unbiased estimator is s2 = ee/(N-K).
“Degrees of freedom correction”
Therefore, the unbiased estimator of 2 is
s2 = ee/(N-K)
7-7/53
Part 7: Estimating the Variance of b
Method 2: Some Matrix Algebra
E[e'e|X ]  2 trace M
What is the trace of M? M is idempotent, so its
trace equals its rank. Its rank equals the number
of nonzero characeristic roots.
Characteric Roots :
Signature of a Matrix = Spectral Decomposition
= Eigen (own) value Decomposition
A = CC' where
C = a matrix of columns such that CC' = C'C = I
 = a diagonal matrix of the characteristic roots
elements of  may be zero
7-8/53
Part 7: Estimating the Variance of b
Decomposing M
Useful Result: If A = CC' is the spectral
decomposition, then A 2  C2 C ' (just multiply)
M = M2 , so  2  . All of the characteristic
roots of M are 1 or 0. How many of each?
trace(A ) = trace(CC')=trace(C'C)=trace( )
Trace of a matrix equals the sum of its characteristic
roots. Since the roots of M are all 1 or 0, its trace is
just the number of ones, which is N-K as we saw.
7-9/53
Part 7: Estimating the Variance of b
Example: Characteristic Roots of a
Correlation Matrix
7-10/53
Part 7: Estimating the Variance of b
R = CΛC   i 1 i cici
6
7-11/53
Part 7: Estimating the Variance of b
Gasoline Data
7-12/53
Part 7: Estimating the Variance of b
X’X and its Roots
7-13/53
Part 7: Estimating the Variance of b
Var[b|X]
Estimating the Covariance Matrix for b|X
The true covariance matrix is 2 (X’X)-1
The natural estimator is s2(X’X)-1
“Standard errors” of the individual coefficients are
the square roots of the diagonal elements.
7-14/53
Part 7: Estimating the Variance of b
X’X
(X’X)-1
s2(X’X)-1
7-15/53
Part 7: Estimating the Variance of b
Standard Regression Results
---------------------------------------------------------------------Ordinary
least squares regression ........
LHS=G
Mean
= 226.09444
Standard deviation
=
50.59182
Number of observs.
=
36
Model size
Parameters
=
7
Degrees of freedom
=
29
Residuals
Sum of squares
= 778.70227
Standard error of e =
5.18187 <= sqr[778.70227/(36 – 7)]
Fit
R-squared
=
.99131
Adjusted R-squared
=
.98951
--------+------------------------------------------------------------Variable| Coefficient
Standard Error t-ratio P[|T|>t]
Mean of X
--------+------------------------------------------------------------Constant|
-7.73975
49.95915
-.155
.8780
PG|
-15.3008***
2.42171
-6.318
.0000
2.31661
Y|
.02365***
.00779
3.037
.0050
9232.86
TREND|
4.14359**
1.91513
2.164
.0389
17.5000
PNC|
15.4387
15.21899
1.014
.3188
1.67078
PUC|
-5.63438
5.02666
-1.121
.2715
2.34364
PPT|
-12.4378**
5.20697
-2.389
.0236
2.74486
--------+-------------------------------------------------------------
7-16/53
Part 7: Estimating the Variance of b
Bootstrapping
Some assumptions that underlie it - the sampling mechanism
Method:
1. Estimate using full sample: --> b
2. Repeat R times:
Draw N observations from the n, with replacement
Estimate  with b(r).
3. Estimate variance with
V = (1/R)r [b(r) - b][b(r) - b]’
7-17/53
Part 7: Estimating the Variance of b
Bootstrap Application
matr;bboot=init(3,21,0.)$
name;x=one,y,pg$
regr;lhs=g;rhs=x$
calc;i=0$
Proc
regr;lhs=g;rhs=x;quietly$
matr;{i=i+1};bboot(*,i)=b$...
Endproc
exec;n=20;bootstrap=b$
matr;list;bboot' $
7-18/53
Store results here
Define X
Compute b
Counter
Define procedure
… Regression
Store b(r)
Ends procedure
20 bootstrap reps
Display results
Part 7: Estimating the Variance of b
Results of Bootstrap Procedure
--------+------------------------------------------------------------Variable| Coefficient
Standard Error t-ratio P[|T|>t]
Mean of X
--------+------------------------------------------------------------Constant|
-79.7535***
8.67255
-9.196
.0000
Y|
.03692***
.00132
28.022
.0000
9232.86
PG|
-15.1224***
1.88034
-8.042
.0000
2.31661
--------+------------------------------------------------------------Completed
20 bootstrap iterations.
---------------------------------------------------------------------Results of bootstrap estimation of model.
Model has been reestimated
20 times.
Means shown below are the means of the
bootstrap estimates. Coefficients shown
below are the original estimates based
on the full sample.
bootstrap samples have
36 observations.
--------+------------------------------------------------------------Variable| Coefficient
Standard Error b/St.Er. P[|Z|>z]
Mean of X
--------+------------------------------------------------------------B001|
-79.7535***
8.35512
-9.545
.0000
-79.5329
B002|
.03692***
.00133
27.773
.0000
.03682
B003|
-15.1224***
2.03503
-7.431
.0000
-14.7654
--------+-------------------------------------------------------------
7-19/53
Part 7: Estimating the Variance of b
Bootstrap Replications
Full sample result
Bootstrapped sample
results
7-20/53
Part 7: Estimating the Variance of b
OLS vs. Least Absolute Deviations
---------------------------------------------------------------------Least absolute deviations estimator...............
Residuals
Sum of squares
=
1537.58603
Standard error of e =
6.82594
Fit
R-squared
=
.98284
--------+------------------------------------------------------------Variable| Coefficient
Standard Error b/St.Er. P[|Z|>z]
Mean of X
--------+------------------------------------------------------------|Covariance matrix based on
50 replications.
Constant|
-84.0258***
16.08614
-5.223
.0000
Y|
.03784***
.00271
13.952
.0000
9232.86
PG|
-17.0990***
4.37160
-3.911
.0001
2.31661
--------+------------------------------------------------------------Ordinary
least squares regression ............
Residuals
Sum of squares
=
1472.79834
Standard error of e =
6.68059
Standard errors are based on
Fit
R-squared
=
.98356
50 bootstrap replications
--------+------------------------------------------------------------Variable| Coefficient
Standard Error t-ratio P[|T|>t]
Mean of X
--------+------------------------------------------------------------Constant|
-79.7535***
8.67255
-9.196
.0000
Y|
.03692***
.00132
28.022
.0000
9232.86
PG|
-15.1224***
1.88034
-8.042
.0000
2.31661
--------+-------------------------------------------------------------
7-21/53
Part 7: Estimating the Variance of b
Quantile Regression:
Application of Bootstrap
Estimation
7-22/53
Part 7: Estimating the Variance of b
Quantile Regression





Q(y|x,) = x,  = quantile
Estimated by linear programming
Q(y|x,.50) = x, .50  median regression
Median regression estimated by LAD (estimates same
parameters as mean regression if symmetric conditional
distribution)
Why use quantile (median) regression?



7-23/53
Semiparametric
Robust to some extensions (heteroscedasticity?)
Complete characterization of conditional distribution
Part 7: Estimating the Variance of b
Estimated Variance for
Quantile Regression
7-24/53

Asymptotic Theory

Bootstrap – an ideal application
Part 7: Estimating the Variance of b
Asymptotic Theory Based Estimator of Variance of Q - REG
Model : yi  βx i  ui , Q( yi | xi , )  βx i , Q[ui | xi , ]  0
Residuals: uˆ  y - βˆ x
i
i
i
1
A 1 C A 1 

N
1
11
N
A = E[fu (0)xx] Estimated by  i 1
1| uˆi | B xi xi
N
B2
Bandwidth B can be Silverman's Rule of Thumb:
Asymptotic Variance:
1.06
 Q(uˆi | .75)  Q(uˆi | .25) 
Min
su ,


.2
N
1.349


(1- )
C = (1- ) E[ xx] Estimated by
XX
N
For  =.5 and normally distributed u, this all simplifies to
 2
1
su  XX  .
2
But, this is an ideal application for bootstrapping.
7-25/53
Part 7: Estimating the Variance of b
 = .25
 = .50
 = .75
7-26/53
Part 7: Estimating the Variance of b
7-27/53
Part 7: Estimating the Variance of b
7-28/53
Part 7: Estimating the Variance of b
Multicollinearity
Not “short rank,” which is a deficiency in the model.
A characteristic of the data set which affects the covariance matrix.
Regardless,  is unbiased. Consider one of the unbiased coefficient estimators
of k. E[bk] = k
Var[b] = 2(X’X)-1 .
The variance of bk is the kth diagonal element of 2(X’X)-1 .
We can isolate this with the result in your text.
Let [X,z] be [Other xs, xk] = [X1,x2]
(a convenient notation for the results in the text).
We need the residual maker, MX.
The general result is that the diagonal element we seek is
[zM1z]-1 , which we know is the reciprocal of the sum of squared residuals
in the regression of z on X.
7-29/53
Part 7: Estimating the Variance of b
I have a sample of 24025 observations in a logit model. Two predictors are highly collinear
(pairwaise corr .96; p<.001); vif are about 12 for eachof them; average vif is 2.63; condition
number is 10.26; determinant of correlation matrix is 0.0211; the two lowest eigen vales are
0.0792 and 0.0427. Centering/standardizing variables does not change the story.
Note: most obs are zeros for these two variables; I only have approx 600 non-zero obs for
these two variables on a total of 24.025 obs.
Both variable coefficients are significant and must be included in the model (as per
specification).
-- Do I have a problem of multicollinearity??
-- Does the large sample size attenuate this concern, even if I have a correlation of .96?
-- What could I look at to ascertain that the consequences of multi-collinearity are not a
problem?
-- Is there any reference I might cite, to say that given the sample size, it is not a problem?
I hope you might help, because I am really in trouble!!!
7-30/53
Part 7: Estimating the Variance of b
Variance of Least Squares
Model : y = Xβ + z + 
1
b
 XX Xz 
Variance of    2 



c
z
X
z
z
 


Variance of c is the lower right element of this matrix.
2
Var[c] =  [zM X z ] 
z * z *
where z* = the vector of residuals from the regression of z on X.
z * z *
The R 2 in that regression is R 2z|X = 1 , so
n
2
 i1 (zi  z)
2
1
z * z*  1  R 2z|X   i 1 (z i  z) 2 . Therefore,
n
Var[c] =  [zM X z ] 
2
7-31/53
1
2
1  R  
2
z|X
n
i 1
(z i  z) 2
Part 7: Estimating the Variance of b
Multicollinearity
Var[c] =  [zM X z ] 
2
1
2
1  R  
2
z|X
n
2
(z

z)
i
i 1
All else constant, the variance of the coefficient on z rises as the fit
in the regression of z on the other variables goes up. If the fit is
perfect, the variance becomes infinite.
"Detecting" multicollinearity?
Variance inflation factor: VIF(z) =
7-32/53
1
.
2
1  R z|X 
Part 7: Estimating the Variance of b
Gasoline Market
Regression Analysis:
logG versus logIncome, logPG
The regression equation is
logG = - 0.468 + 0.966 logIncome - 0.169 logPG
Predictor
Coef SE Coef
T
P
Constant
-0.46772 0.08649 -5.41 0.000
logIncome
0.96595 0.07529 12.83 0.000
logPG
-0.16949 0.03865 -4.38 0.000
S = 0.0614287
R-Sq = 93.6%
R-Sq(adj) = 93.4%
Analysis of Variance
Source
DF
SS
MS
F
P
Regression
2 2.7237 1.3618 360.90 0.000
Residual Error 49 0.1849 0.0038
Total
51 2.9086
7-33/53
Part 7: Estimating the Variance of b
Gasoline Market
Regression Analysis: logG versus logIncome, logPG, ...
The regression equation is
logG = - 0.558 + 1.29 logIncome - 0.0280 logPG
- 0.156 logPNC + 0.029 logPUC - 0.183 logPPT
Predictor
Coef SE Coef
T
P
Constant
-0.5579
0.5808 -0.96 0.342
logIncome
1.2861
0.1457
8.83 0.000
logPG
-0.02797 0.04338 -0.64 0.522
logPNC
-0.1558
0.2100 -0.74 0.462
logPUC
0.0285
0.1020
0.28 0.781
logPPT
-0.1828
0.1191 -1.54 0.132
S = 0.0499953
R-Sq = 96.0%
R-Sq(adj) = 95.6%
Analysis of Variance
Source
DF
SS
MS
F
P
Regression
5 2.79360 0.55872 223.53 0.000
Residual Error 46 0.11498 0.00250
Total
51 2.90858
The standard error on logIncome doubles when the
three variables are added to the equation.
7-34/53
Part 7: Estimating the Variance of b
Condition Number and
Variance Inflation Factors
Condition number
larger than 30 is
‘large.’
What does this
mean?
7-35/53
Part 7: Estimating the Variance of b
7-36/53
Part 7: Estimating the Variance of b
The Longley Data
7-37/53
Part 7: Estimating the Variance of b
NIST Longley Solution
7-38/53
Part 7: Estimating the Variance of b
Excel Longley Solution
7-39/53
Part 7: Estimating the Variance of b
The NIST Filipelli Problem
7-40/53
Part 7: Estimating the Variance of b
Certified Filipelli Results
7-41/53
Part 7: Estimating the Variance of b
Minitab Filipelli Results
7-42/53
Part 7: Estimating the Variance of b
Stata Filipelli Results
7-43/53
Part 7: Estimating the Variance of b
Even after dropping two (random
columns), results are only correct to 1
or 2 digits.
7-44/53
Part 7: Estimating the Variance of b
Regression of x2 on all other variables
7-45/53
Part 7: Estimating the Variance of b
Using QR Decomposition
7-46/53
Part 7: Estimating the Variance of b
Multicollinearity
There is no “cure” for collinearity. Estimating something else is not helpful
(principal components, for example).
There are “measures” of multicollinearity, such as the condition number of X
and the variance inflation factor.
Best approach: Be cognizant of it. Understand its implications for estimation.
What is better: Include a variable that causes collinearity, or drop the variable
and suffer from a biased estimator?
Mean squared error would be the basis for comparison.
Some generalities. Assuming X has full rank, regardless of the condition,
b is still unbiased
Gauss-Markov still holds
7-47/53
Part 7: Estimating the Variance of b
Specification and Functional Form:
Nonlinearity
Population
Estimators
y  1  2 x  3 x 2  4 z  
yˆ  b1  b2 x  b3 x 2  b4 z
E[ y | x, z ]
x 
 2  23 x
x
ˆ x  b2  2b3 x
Estimator of the variance of ˆ x
Est.Var[ˆ x ]  Var[b2 ]  4 x 2Var[b3 ]  4 xCov[b2 , b3 ]
7-48/53
Part 7: Estimating the Variance of b
Log Income Equation
---------------------------------------------------------------------Ordinary
least squares regression ............
LHS=LOGY
Mean
=
-1.15746
Estimated Cov[b1,b2]
Standard deviation
=
.49149
Number of observs.
=
27322
Model size
Parameters
=
7
Degrees of freedom
=
27315
Residuals
Sum of squares
=
5462.03686
Standard error of e =
.44717
Fit
R-squared
=
.17237
--------+------------------------------------------------------------Variable| Coefficient
Standard Error b/St.Er. P[|Z|>z]
Mean of X
--------+------------------------------------------------------------AGE|
.06225***
.00213
29.189
.0000
43.5272
AGESQ|
-.00074***
.242482D-04
-30.576
.0000
2022.99
Constant|
-3.19130***
.04567
-69.884
.0000
MARRIED|
.32153***
.00703
45.767
.0000
.75869
HHKIDS|
-.11134***
.00655
-17.002
.0000
.40272
FEMALE|
-.00491
.00552
-.889
.3739
.47881
EDUC|
.05542***
.00120
46.050
.0000
11.3202
--------+------------------------------------------------------------Average Age = 43.5272. Estimated Partial effect = .066225 – 2(.00074)43.5272 = .00018.
Estimated Variance 4.54799e-6 + 4(43.5272)2(5.87973e-10) + 4(43.5272)(-5.1285e-8)
= 7.4755086e-08.
Estimated standard error = .00027341.
7-49/53
Part 7: Estimating the Variance of b
Specification and Functional Form:
Interaction Effect
Population
y  1  2 x  3 z  4 xz  
Estimators
yˆ  b1  b2 x  b3 z  b4 xz
E[ y | x, z ]
x 
 2  4 z
ˆ x  b2  b4 z
x
Estimator of the variance of ˆ x
Est.Var[ˆ x ]  Var[b2 ]  z 2Var[b4 ]  2 zCov[b2 , b4 ]
7-50/53
Part 7: Estimating the Variance of b
Interaction Effect
---------------------------------------------------------------------Ordinary
least squares regression ............
LHS=LOGY
Mean
=
-1.15746
Standard deviation
=
.49149
Number of observs.
=
27322
Model size
Parameters
=
4
Degrees of freedom
=
27318
Residuals
Sum of squares
=
6540.45988
Standard error of e =
.48931
Fit
R-squared
=
.00896
Adjusted R-squared
=
.00885
Model test
F[ 3, 27318] (prob) =
82.4(.0000)
--------+------------------------------------------------------------Variable| Coefficient
Standard Error b/St.Er. P[|Z|>z]
Mean of X
--------+------------------------------------------------------------Constant|
-1.22592***
.01605
-76.376
.0000
AGE|
.00227***
.00036
6.240
.0000
43.5272
FEMALE|
.21239***
.02363
8.987
.0000
.47881
AGE_FEM|
-.00620***
.00052
-11.819
.0000
21.2960
--------+------------------------------------------------------------Do women earn more than men (in this sample?) The +.21239 coefficient on FEMALE would
suggest so. But, the female “difference” is +.21239 - .00620*Age. At average Age, the
effect is .21239 - .00620(43.5272) = -.05748.
7-51/53
Part 7: Estimating the Variance of b
7-52/53
Part 7: Estimating the Variance of b
7-53/53
Part 7: Estimating the Variance of b