Transcript lecture note 2
STAT 497
LECTURE NOTES 2 1
THE AUTOCOVARIANCE AND THE AUTOCORRELATION FUNCTIONS • For a stationary process
{Y t } ,
autocovariance between
Y t
the and
Y t-k
is
k
Cov
Y t
,
Y t
k
E
Y t
Y t
k
and the autocorrelation function is
k
Corr
Y t
,
Y t
k
k
0
ACF
2
THE AUTOCOVARIANCE AND THE AUTOCORRELATION FUNCTIONS
PROPERTIES: 1.
0
Var
t
2.
3.
k k
0
k
0 and
k
k
1 .
k
1 .
,
k
.
4. (necessary condition)
k
definite
i n n
1
i
j
t i
and
k
t j
0 are positive semi-
i n n
1
i
j
t i
t j
0 for any set of time points
t 1 ,t 2 ,…,t n
and any real numbers 1 , 2 ,…, 3 n .
THE PARTIAL AUTOCORRELATION FUNCTION (PACF) • • PACF is the correlation between
Y t
and
Y t-k
their mutual linear dependency on the intervening variables
Y t-1 , Y t-2 , …, Y t-k+1
been removed.
has after The conditional correlation
Corr
Y t
,
Y t
k Y t
1 ,
Y t
2 , ,
Y t
k
1
kk
is usually referred as the partial autocorrelation in time series.
e
.
g
., 11 22
Corr Corr
Y t
Y t
,
Y t
,
Y t
1 2
Y t
1 1 4
CALCULATION OF PACF
1. REGRESSION APPROACH:
Consider a model
Y t
k
k
1
Y t
k
1
k
2
Y t
k
2
kk Y t
e t
k
from a zero mean stationary process where
ki
denotes the coefficients of
Y t
k+i
and
e t
k
is the zero mean error term which is uncorrelated with
Y t
k+i , i=0,1,…,k
.
• Multiply both sides by
Y t
k+j Y t
k Y t
k
j
k
1
Y t
k
1
Y t
k
j
kk Y t Y t
k
j
e t
k Y t
k
j
5
CALCULATION OF PACF
and taking the expectations
j
k
1
j
1 diving both sides by 0
k
2
j
2
kk
j
k
j
k
1
j
1
k
2
j
2
kk
j
k
PACF 6
CALCULATION OF PACF
• For
j=1,2,…,k,
we have the following system of equations 1 2
k
1
k
1 1
k
2
k
1 2
kk
kk
k k
1 2
k
k
1
k
1
k
2
k
2
kk
7
CALCULATION OF PACF
• Using Cramer’s rule successively for
k=1,2,…
11 1 22 1 1 1 1 1 2 1 1 1 2 1 2 1 2 8
CALCULATION OF PACF
kk
1 1
k
1 1 1
k
1 1 1 1
k
2 1
k
2 2 1 2
k
1 3
k
3
k
k
2 3 1
k
2
k
3 1 1 2
k k
1
k
2 1 9
CALCULATION OF PACF
2. Levinson and Durbin’s Recursive Formula:
kk
1
k
j k
1 1
k j k
1 1
k
1 , 1 ,
j j
k k
j j where
kj
k
1 ,
j
kk
k
1 ,
k
j
,
j
1 , 2 , ,
k
1 .
10
WHITE NOISE (WN) PROCESS
• A process
{a t }
is called a white noise (WN) process, if it is a sequence of uncorrelated random variables from a fixed distribution with constant mean variance all
k≠0
.
{Var(a t
2
{E(a t )=
and
},
constant
Cov(Y t , Y t-k )=0
for
Y t
a t
11
WHITE NOISE (WN) PROCESS
• It is a stationary process with autocovariance function
k
2
a
,
k
0 ,
k
0 0
ACF PACF
k
1 , 0 ,
k k
0 0
kk
1
,
0
, k k
0 0
Basic Phenomenon:
ACF=PACF=
0
,
k
0
.
12
WHITE NOISE (WN) PROCESS
• • • • White noise (in spectral analysis): white light is produced in which all frequencies (i.e., colors) are present in equal amount.
Memoryless process Building block from which we can construct more complicated models It plays the role of an orthogonal basis in the general vector and function analysis.
13
ESTIMATION OF THE MEAN, AUTOCOVARIANCE AND AUTOCORRELATION • THE SAMPLE MEAN:
y
t n
1
n y t with E
and Var
0
n k
n
1
n
1
k n
k
.
Because Var
n
0 ,
Y is a CE for
.
n
lim
Y
if this holds, the process is ergodic for the mean.
in mean square
14
ERGODICITY
• • Kolmogorov’s law of large number (LLN) tells that if
X i
i.i.d.(μ,
2 )
for
i = 1, . . . , n
, then we have the following limit for the ensemble average
Y n
i n
1
Y i
.
n
In time series, we have time series average, not ensemble average. Hence, the mean is computed by averaging over time. Does the time series average converges to the same limit as the ensemble average? The answer is yes, if
Y t
is stationary and ergodic.
15
ERGODICITY
• • A covariance stationary process is said to ergodic for the mean, if the time series average converges to the population mean.
Similarly, if the sample average provides an consistent estimate for the second moment, then the process is said to be ergodic for the second moment.
16
ERGODICITY
• A sufficient condition for a covariance stationary process to be ergodic for the mean is that . Further, if the process is
k
0 Gaussian, then absolute summable autocovariances also ensure that the process is ergodic for all moments.
17
THE SAMPLE AUTOCOVARIANCE FUNCTION ˆ
k
1
n t n
k
1
Y t
Y
Y t
k
Y
or ˆ
k
n
1
k t n
k
1
Y t
Y
Y t
k
Y
18
THE SAMPLE AUTOCORRELATION FUNCTION ˆ
k
r k
t n
k
1
Y t n
A plot versus
k t
1
Y Y t
Y t Y
k
2
Y
,
k
0 , 1 , 2 ,...
a sample correlogram • • distributed with mean
k
and variance is approximated by Bartlett’s approximation for processes in which
k
=
0
for
k>m
.
19
THE SAMPLE AUTOCORRELATION FUNCTION •
Var
1 1 2 1 2 2 2 2 2 2
m
n
In practice,
i
’s are unknown and replaced by their sample estimates, . Hence, we have the
s k
1
n
1 2 1 2 2 2 2 2 2
m
20
• • THE SAMPLE AUTOCORRELATION FUNCTION For a WN process, we have
s
ˆ
k
1
n
The ~95% confidence interval for
k
: ˆ
k
2 1
n
For a WN process, it must be close to zero.
• Hence, to test the process is WN or not, draw a
2/n 1/2
lines on the sample correlogram. If all ˆ
k
are inside the limits, the process could be WN (we need to check the sample PACF, too). 21
THE SAMPLE PARTIAL AUTOCORRELATION FUNCTION ˆ 11 ˆ 1 ˆ
kk
ˆ
k
1
j k
1 1 ˆ
k j k
1 1 ˆ
k
1 , 1 ,
j j
ˆ
k
ˆ
k
j j where
ˆ
kj
ˆ
k
1 ,
j
ˆ
kk
ˆ
k
1 ,
k
j
,
j
1 , 2 , ,
k
1 .
• • For a WN process,
Var
kk
1
n
2/n 1/2
can be used as critical limits on
kk
to test the hypothesis of a WN process.
22
BACKSHIFT (OR LAG) OPERATORS
• Backshift operator,
B
is defined as
B j Y t
Y t
j
,
j
0
with B 0 BY t
Y t
1 1 .
B
2
Y t
Y t
2
B
12
Y t
Y t
12 e.g. Random Shock Process:
Y t
Y t
1
e t Y t
Y t
1
Y t
1
B BY t
Y t
e t e t e t
23
• • MOVING AVERAGE REPRESENTATION OF A TIME SERIES Also known as Random Shock Form or Wold (1938) Representation.
Let
{Y t }
be a time series. For a stationary process
{Y t },
we can write
{Y t }
as a linear combination of sequence of uncorrelated (WN) r.v.s.
A GENERAL LINEAR PROCESS:
Y t
a t
1
a t
where
0 =I, {a t }
is a
0
1 2
a t
2
j
0 mean WN process and
j
0
j a t
2
j
j
.
24
MOVING AVERAGE REPRESENTATION OF A TIME SERIES
Y t
a t
1
Ba t
2
B
2
a t
1 1
B
a t
2
B
2
where
a t
1 1
B
j
0
j B j a t
2
B
2
j
0
j B j
25
MOVING AVERAGE REPRESENTATION OF A TIME SERIES
E
E
a t
1
a t
1 2
a t
2
....
k
2
a
k Var k a t
k E
Y t
k
0 1
a t
Y t k
2
a
k
1
j
0
...
j
2
a t
k
1
k
1 2
a
2
k
2 2
a
...
1
a t
k
1 2
a
i
0
i k
i
k
i
i j
0 0
k j
2
i
2
a t
k
2
....
26
MOVING AVERAGE REPRESENTATION OF A TIME SERIES • • • Because they involve infinite sums, to be statinary
k
j
Y t
0
j
2
Y t
Cauchy
Schwarz Inequality
the process to be stationary.
Y t k
1 / 2
a
2
j
0
j
2 It is a non-deterministic process: A process contains no deterministic components (no randomness in the future states of the system) that can be forecast exactly from its own past.
27
• AUTOCOVARIANCE GENERATING FUNCTION For a given sequence of autocovariances
k
,
k=0,
1,
2,…
the autocovariance generating function is defined as
k
k B k
where the variance of a given process
0
is the coefficient of
k
,
B 0
and the autocovariance of lag is the coefficient of both
B k
and
B
k
.
k
2
B
2 1
B
1 0 1
B
2
B
2 2 1 28
• AUTOCOVARIANCE GENERATING FUNCTION Using and stationarity
i
0
i i
2
a k
k
i
0
i
k
B k j
i
k
2
a j
0
i
0
i
j
B j
i
2
a
2
a i
0 where
j =0
for
j<0
.
i B
i j
0
j B j
29
AUTOCORRELATION GENERATING FUNCTION
k
k B k
0 30
EXAMPLE
Y t
Y t
1
a t where
1
and a t
~
iid
a
.
a) Write the above equation in random shock form.
b) Find the autocovariance generating function.
31
AUTOREGRESSIVE REPRESENTATION OF A TIME SERIES • This representation is also known as INVERTED FORM.
• Regress the value of
Y t
at time
t
past plus a random shock.
Y t
1
Y t
1 2
B
2
Y t
Y t
2 on its own
a t
a t
j
0
j B j
Y t
a t with
0
1
and 1
j
1
j
.
32
AUTOREGRESSIVE REPRESENTATION OF A TIME SERIES • • • It is an invertible process (it is important for forecasting). Not every stationary process is invertible (Box and Jenkins, 1978).
Invertibility provides uniqueness of the autocorrelation function.
It means that different time series models can be re-expressed by each other.
33
INVERTIBILITY RULE USING THE RANDOM SHOCK FORM • For a linear process,
Y t
a t
to be invertible, the roots of
(B)=0
as a function of
B
must lie outside the unit circle. • If is a root of
(B)
, then
|
|>1 .
(real number)
|
| is the absolute value of .
(complex number)
c
id
|
| is
c
2
d
2 .
34
INVERTIBILITY RULE USING THE RANDOM SHOCK FORM • It can be stationary if the process can be re written in a RSF, i.e.,
Y t
1
a t
a t
1
where j
0
j 2
.
35
STATIONARITY RULE USING THE INVERTED FORM • • For a linear process,
Y t
a t
to be invertible, the roots of
(B)=0
as a function of
B
If must lie outside the unit circle.
is a root of (B), then |
|>1.
36
RANDOM SHOCK FORM AND INVERTED FORM • AR and MA representations are not the model form. Because they contain infinite number of parameters that are impossible to estimate from a finite number of observations.
37
TIME SERIES MODELS
• In the Inverted Form of a process, if only finite number of weights are non-zero, i.e., 1 1 , 2 2 , ,
p
p
and
Π k
0 ,
k
p
, the process is called AR(p) process. 38
TIME SERIES MODELS
• In the Random Shock Form of a process, if only finite number of weights are non-zero, i.e., 1 1 , 2 2 , ,
q
q
and
k
0 ,
k
q
, the process is called MA(q) process.
39
TIME SERIES MODELS
• AR(p) Process:
Y t
1
Y t
1
p
Y t
p
a t Y t
c
1
Y t
1
p Y t
p
a t where
1 -
1
c
p .
• MA(q) Process:
Y t
a t
1
a t
1
q a t
q
.
40
TIME SERIES MODELS
• The number of parameters in a model can be large. A natural alternate is the mixed AR and MA process ARMA(p,q) process
Y t
1
c
1
B
1
Y t
1
p B p
Y t p Y t
c p
1
a t
θ
1
θ
1
a t B
1
θ q
B θ q q a t-q
a t
• For a fixed number of observations, the more parameters in a model, the less efficient is the estimation of the parameters. Choose a simpler model to describe the phenomenon.
41