ECONOMETRICS I ECON GR5411
Lecture 16 – Generalized Least Squares Regression Model
by
Seyhan Erden Columbia University
Generalized Least Squares Regression: Recall that when 𝜀|𝑋 is not spherical, the model
is
𝑦 = 𝑋𝛽 + 𝜀 𝐸𝜀|𝑋 =0
𝑉𝑎𝑟 𝜀|𝑋 =𝐸 𝜀𝜀′|𝑋 =Ω
where Ω is an 𝑛×𝑛 positive definite matrix,likely to be a function of 𝑋. When errors are spherical we have the special case that Ω = 𝜎2𝐼
11/11/20 Lecture 16 GR5411 by Seyhan Erden 2
As a covariance matrix, Ω is necessarily symmetric and positive semi-definite.
Two leading cases for Ω ≠ 𝜎2𝐼 are heteroskedasticity and autocorrelation.
11/11/20 Lecture 16 GR5411 by Seyhan Erden 3
Heteroskedasticity arises when errors do not have the same variances. This can happen with cross section as well as with time series data. For example volatile high-frequency data such as daily observations of financial market and in cross section data where the scale of observations depend on the level of the regressor. Disturbances are still assumed to be uncorrelated across observations under heteroskedasticity so Ω would be
𝜎52 0 0 0 0 𝜎2 … ⋮
0⋮ 0⋮ ⋱0𝜎92
Ω=
11/11/20 Lecture 16 GR5411 by Seyhan Erden 4
Generalized Least Squares (GLS):
Ω can be (and usually is) a function of 𝑋 matrix.
For this reason it is sometimes written as Ω(𝑋) and under heteroskedastic errors, in which case Ω(𝑋) is a diagonal matrix with diagonal elements 𝜆h 𝑋> , where 𝜆 is a constant and h is a function. For example Weighted Least Squares (the special case of GLS) 𝜆h 𝑋> = 𝑣𝑎𝑟(𝜀>|𝑋>)
11/11/20 Lecture 16 GR5411 by Seyhan Erden
5
Then we can write Ω as under heteroskedasticity and autocorrelation
𝜎52 𝜌…𝜌9A5
Ω= 𝜌⋮ ⋱
𝜌9A5 … 𝜎92
Off-diagonal values depend on the model, remember this Ω is for a heteroskedastic time- series setting, when errors follow serial correlation of unknown form.
11/11/20 Lecture 16 GR5411 by Seyhan Erden 6
𝜌9A2
⋮
Autoregressive Processes:
An AR(1) series can be expressed as
𝜀B = 𝜌𝜀BA5 + 𝑣B assuming weak stationarity, 𝑣B~𝑁(0, 𝜎2)
If we assume 𝜀B is stationary, then the mean of𝜀B 𝐸𝜀B =𝐸𝜀BA5 =𝜇
hence,
𝐸𝜀B =𝜌𝐸𝜀BA5 +𝐸𝑣B
𝜇=0=0 1−𝜌
11/11/20
Lecture 16 GR5411 by Seyhan Erden 7
Autoregressive Processes:
Variance of 𝜀B is
𝑉𝑎𝑟 𝜀B = 𝜌2𝑉𝑎𝑟(𝜀BA5) + 𝑉𝑎𝑟(𝑣B)
Since 𝑉𝑎𝑟 𝜀B = 𝑉𝑎𝑟(𝜀BA5) and 𝑉𝑎𝑟 𝑣B = 𝜎2 𝑉𝑎𝑟𝜀B =𝛾J= 𝜎2
1−𝜌2
11/11/20 Lecture 16 GR5411 by Seyhan Erden
8
Autoregressive Processes:
Covariance between 𝜀B and 𝜀BA5 is 𝐶𝑜𝑣𝜀B,𝜀BA5 =𝐸 𝜀B−0 𝜀BA5−0
We need to use the fact that 𝐴𝑅 1 = 𝑀𝐴(∞) here for convenience. Note that
𝜀B = 𝜌𝜀BA5 + 𝑣B
𝜀BA5 = 𝜌𝜀BA2 + 𝑣BA5 𝜀BA2 = 𝜌𝜀BAQ + 𝑣BA2
11/11/20 Lecture 16 GR5411 by Seyhan Erden 9
Autoregressive Processes:
Pluging in 𝜀BA> iteratively 𝜀B=𝜌𝜌𝜀BA2+𝑣BA5 +𝑣B
𝜀B = 𝜌2𝜀BA2 + 𝜌𝑣BA5 + 𝑣B
𝜀B =𝜌2 𝜌𝜀BAQ +𝑣BA2 +𝜌𝑣BA5 +𝑣B
𝜀B = 𝜌Q𝜀BAQ + 𝜌2𝑣BA2 + 𝜌𝑣BA5 + 𝑣B
𝜀B = 𝜌R𝜀BAR + 𝜌Q𝑣BAQ + 𝜌2𝑣BA2 + 𝜌𝑣BA5 + 𝑣B 𝜀B = 𝑣B +𝜌𝑣BA5 +𝜌2𝑣BA2 +𝜌Q𝑣BAQ +⋯
11/11/20 Lecture 16 GR5411 by Seyhan Erden 10
Covariance of AR(1) model:
Now, we can easily find the covariance of an 𝐴𝑅(1) process:
𝐶𝑜𝑣𝜀B,𝜀BA5 =𝐸 𝜀B−0 𝜀BA5−0
Let
𝛾5 =𝐶𝑜𝑣 𝜀B,𝜀BA5 𝛾2 =𝐶𝑜𝑣 𝜀B,𝜀BA2 𝛾Q =𝐶𝑜𝑣 𝜀B,𝜀BAQ
⋮
𝛾9 =𝐶𝑜𝑣 𝜀B,𝜀BA9
11/11/20
Lecture 16 GR5411 by Seyhan Erden 11
Thus,
First covariance of AR(1):
𝛾5=𝐸 𝜀B−0 𝜀BA5−0
= 𝐸 𝜀B𝜀BA5
= 𝜌𝐸 𝜀BA5𝜀BA5 + 𝐸 𝑣B𝜀BA5 = 𝜌𝛾J
𝛾5=𝜌 𝜎2 1−𝜌2
𝛾2=𝐸 𝜀B−0 𝜀BA2−0
=𝜌𝐸𝜀BA5𝜀BA2 +𝐸𝑣B𝜀BA2 = 𝜌𝛾5 𝜎2
=𝜌 𝜌1−𝜌2
𝛾2 = 𝜌2 𝜎2 1−𝜌2
Thus,
11/11/20
Lecture 16 GR5411 by Seyhan Erden 12
𝛾Q=𝐸 𝜀B−0 𝜀BAQ−0
= 𝐸 𝜀B𝜀BAQ
= 𝜌𝐸 𝜀BA5𝜀BAQ = 𝜌𝛾2 𝜎2
=𝜌 𝜌21−𝜌2
𝛾Q = 𝜌Q 𝜎2 1−𝜌2
𝛾T=𝐸 𝜀B−0 𝜀BAT−0
= 𝜌𝐸 𝜀BA5𝜀BAT + 𝐸 𝑣B𝜀BAT = 𝜌𝛾TA5 𝜎2
=𝜌 𝜌TA51−𝜌2
𝛾T=𝜌T 𝜎2 1−𝜌2
+ 𝐸 𝑣B𝜀BAQ
11/11/20
Lecture 16 GR5411 by Seyhan Erden 13
11/11/20
Covariance of AR(1):
𝛾5=𝜌 𝜎2 1−𝜌2
𝛾2 = 𝜌2 𝜎2 1−𝜌2
𝛾Q = 𝜌Q 𝜎2 1−𝜌2
. . .
𝛾9A5 = 𝜌9A5 𝜎2 1−𝜌2
𝛾9=𝜌9 𝜎2 1−𝜌2
Lecture 16 GR5411 by Seyhan Erden 14
Then we can write Ω as
𝜎2 1𝜌…𝜌9A5
Ω= 𝜌⋮⋱𝜌9A2 1−𝜌2 ⋮
𝜌9A5 … 1
Off-diagonal values depend on the model, remember this Ω is for a heteroskedastic time- series setting, when errors follow an AR(1)process such as 𝜀B = 𝜌𝜀BA5 + 𝑣B
11/11/20 Lecture 16 GR5411 by Seyhan Erden 15
Or we can write Ω as
1 𝜌… 𝜌9A5
Ω=𝛾J 𝜌⋮ ⋱ 𝜌9A2
9A5 ⋮ 𝜌…1
Off-diagonal values depend on the model, remember this Ω is for a heteroskedastic time- series setting, when errors follow an AR(1)process such as 𝜀B = 𝜌𝜀BA5 + 𝑣B
11/11/20 Lecture 16 GR5411 by Seyhan Erden 16
Recall: Ordinary Least Squares (OLS):
The essential results for classical model with spherical errors, that is when 𝜀|𝑋 is spherical, the model is
𝑦 = 𝑋𝛽 + 𝜀
𝐸𝜀|𝑋 =0 2
𝑉𝑎𝑟 𝜀|𝑋 =𝐸 𝜀𝜀′|𝑋 =𝜎 𝐼
OLS estimator
𝛽U= 𝑋V𝑋A5𝑋V𝑦=𝛽+ 𝑋V𝑋A5𝑋V𝜀
is the best linear unbiased (BLU), consistent and asymptotically normally distributed (CAN) and if errors are normally distributed, asymptotically efficient among all CAN estimators.
11/11/20 Lecture 16 GR5411 by Seyhan Erden 17
Generalized Least Squares (GLS):
However, if the model is
𝑦 = 𝑋𝛽 + 𝜀 𝐸𝜀|𝑋 =0
𝑉𝑎𝑟 𝜀|𝑋 =𝐸 𝜀𝜀′|𝑋 =Ω
OLS remains to be unbiased, consistent and asymptotically normally distributed, however, it will no longer be efficient. And, the usual inference procedures are no longer appropriate.
11/11/20 Lecture 16 GR5411 by Seyhan Erden 18
Generalized Least Squares:
Recall that when 𝜀|𝑋 is not spherical, the model is 𝑦 = 𝑋𝛽 + 𝜀
𝐸𝜀|𝑋 =0
𝑉𝑎𝑟 𝜀|𝑋 =𝐸 𝜀𝜀′|𝑋 =Ω
where Ω is positive definite. When errors are spherical we have the special case that Ω = 𝜎2𝐼
Instead of fixing up the standard errors, we can efficiently estimate the parameters. Let
Ω = 𝐶Λ𝐶′
Where columns of 𝐶 are characteristic vectors of Ω and the characteristic roots of Ω are arrayed in the diagonal matrix Λ.
Λ = Λ5/2Λ5/2
11/11/20 Lecture 16 GR5411 by Seyhan Erden 19
Generalized Least Squares (GLS):
and
5/2 𝜆5 ⋯ 0 Λ=⋮⋱⋮
0 ⋯ 𝜆9
Then, we can factorize Ω and write Ω = 𝐶Λ𝐶V
where 𝑇 = 𝐶Λ5/2 11/11/20
= 𝐶Λ5/2Λ5/2𝐶V
=𝐶Λ5/2 𝐶Λ5/2 ′ = 𝑇𝑇′
Lecture 16 GR5411 by Seyhan Erden 20
Generalized Least Squares (GLS):
Then, let 𝑃′ = 𝐶ΛA5/2, hence ΩA5 = 𝑃V𝑃
Transform the model by pre multiplying by 𝑃, 𝑦 = 𝑋𝛽 + 𝜀
𝑃𝑦 = 𝑃𝑋𝛽 + 𝑃𝜀 𝑦∗= 𝑋∗𝛽 + 𝜀∗
Since,
𝐸 𝜀∗𝜀∗V|𝑋∗ = 𝐸 𝑃𝜀 𝜀V𝑃V|𝑋∗ = 𝑃𝜎2Ω𝑃V = 𝜎2𝐼
We can apply the classical model to this transformed model.
11/11/20 Lecture 16 GR5411 by Seyhan Erden 21
Generalized Least Squares (GLS):
We assume that Ω is known for now, thus 𝑦∗ and 𝑋∗ are observed data
Then, OLS estimator
𝛽U =(𝑋V𝑋)A5𝑋𝑦 ]^_ ∗∗ ∗∗
= (𝑋V𝑃V𝑃𝑋)A5𝑋V𝑃V𝑃𝑦 = (𝑋VΩA5𝑋)A5𝑋VΩA5𝑦
This estimator is the generalized least squares (GLS) estimator.
11/11/20 Lecture 16 GR5411 by Seyhan Erden 22
Properties of GLS:
GLS estimator is unbiased: If 𝐸 𝜀∗|𝑋∗ = 0, then
𝐸𝛽U 𝑋=𝐸(𝑋V𝑋)A5𝑋𝑦|𝑋 ]^_∗ ∗∗∗∗∗
= 𝛽 + 𝐸 (𝑋∗V𝑋∗)A5𝑋∗ 𝜀∗|𝑋∗ =𝛽
GLS estimator is consistent:
If 𝑝𝑙𝑖𝑚 95 𝑋∗V𝑋∗ = 𝑄∗ is positive definite and
finite.
11/11/20 Lecture 16 GR5411 by Seyhan Erden 23
Properties of GLS:
GLS estimator is asymptotically distributed, with mean 𝛽 and sampling variance
𝑉𝑎𝑟𝛽U |𝑋 =𝐸(𝛽U −𝛽)(𝛽U −𝛽)′𝑋
]^_ ∗
]^_ ]^_ ∗
= (𝑋∗V𝑋∗)A5𝑋∗𝐸 𝜀∗𝜀∗V|𝑋∗ 𝑋∗V(𝑋∗V𝑋∗)A5
= 𝜎2(𝑋∗V𝑋∗)A5
A5
= 𝜎2 𝑋V𝑃V𝑃𝑋 A5 = 𝜎2 𝑋VΩA5𝑋 A5
=𝜎2 𝑃𝑋V𝑃𝑋
11/11/20
Lecture 16 GR5411 by Seyhan Erden 24
Finite Sample Properties of OLS in Generalized Regression Model:
Under Generalized regression model, OLS estimator is asymptotically distributed, with mean 𝛽 and sampling variance
𝑉𝑎𝑟𝛽U|𝑋 =𝐸(𝛽U−𝛽)(𝛽U−𝛽)′𝑋
= (𝑋V𝑋)A5𝑋V𝐸 𝜀𝜀′|𝑋 𝑋 𝑋V𝑋 A5
= (𝑋V𝑋)A5𝑋V𝜎2Ω 𝑋(𝑋V𝑋)A5
𝜎21A51 1A5 = 𝑛 𝑛𝑋V𝑋 𝑛𝑋VΩ𝑋 𝑛𝑋V𝑋
If the regressors are stochastic then the unconditional variance is 𝐸e 𝑉𝑎𝑟 𝛽f|𝑋 and if 𝜀 is normally distributed, then
𝛽U|𝑋 ~𝑁 𝛽, 𝜎2(𝑋V𝑋)A5 𝑋VΩA5𝑋 (𝑋V𝑋)A5
11/11/20 Lecture 16 GR5411 by Seyhan Erden 25
Non-spherical errors:
When the errors are non-spherical, 𝑉𝑎𝑟(𝛽U) =
𝜎2 𝑋V𝑋 A5 will give wrong inference (rejecting the null hypothesis too often or not often enough)
Because U 2 V A5 V V A5 𝑉𝑎𝑟𝛽𝑋=𝜎𝑋𝑋 𝑋Ω𝑋𝑋𝑋
U 𝜎21 A51 1
If 𝑄gg ≡ 𝑝𝑙𝑖𝑚 95 𝑋V𝑋 and 𝑝𝑙𝑖𝑚 95 𝑋VΩ𝑋 are both
A5 𝑉𝑎𝑟𝛽𝑋=𝑛𝑛𝑋V𝑋 𝑛𝑋VΩ𝑋 𝑛𝑋V𝑋
positive definite, 𝛽U is also consistent.
11/11/20 Lecture 16 GR5411 by Seyhan Erden 26
Non-spherical errors:
The asymptotic variance is
𝑎𝑠𝑦𝑚.𝑣𝑎𝑟(𝛽U)=𝑛1𝑄A5𝑝𝑙𝑖𝑚 𝑛1𝑋V𝜎2Ω𝑋 𝑄A5
To construct this variance, we need to estimate
𝑛1 𝑋 V Ω 𝑋 whereΩ= 𝜎>T
11/11/20 Lecture 16 GR5411 by Seyhan Erden 27
White’s Heteroskedasticity Robust Estimator:
The asymptotic (sandwich) variance is
𝑎𝑠𝑦𝑚. 𝑣𝑎𝑟(𝛽U) = 1 𝑄A5𝑝𝑙𝑖𝑚 1 𝑋VΩ𝑋 𝑄A5 𝑛gg 𝑛 gg
Underheteroskedasticity,Ω=𝑑𝑖𝑎𝑔 𝜎>2 =𝑑𝑖𝑎𝑔 𝜎52,…,𝜎92 White’s variance estimator
A5 lUV2VV
11A519 1
𝑎𝑠𝑦𝑚.𝑣𝑎𝑟(𝛽)= 𝑋𝑋 𝑛𝑛𝑛>>𝑛
which is robust to heteroskedasticity of unknown form.
11/11/20 Lecture 16 GR5411 by Seyhan Erden 28
m𝜀̂ 𝑥>𝑥 𝑋𝑋 >n5
The covariance 𝐶𝑜𝑣 𝛽Uqrs, 𝛽Uqrs − 𝛽Utrs , of the GLS estimator 𝛽Uqrs = 𝑋′ΩA5𝑋 A5𝑋VΩA5𝑦 ; and the OLS estimator, 𝛽Utrs =
𝑋′𝑋 A5𝑋V𝑦?
𝛽U = 𝑋′ΩA5𝑋 A5𝑋VΩA5𝑦 = 𝑋′ΩA5𝑋 A5𝑋VΩA5 𝑋𝛽 + 𝜀
Solution:
]^_ = 𝛽 + 𝑋′ΩA5𝑋 A5𝑋VΩA5𝜀
𝛽U −𝛽=𝑋′ΩA5𝑋A5𝑋VΩA5𝜀 ]^_
𝛽Uu^_ = 𝑋′𝑋 A5𝑋V𝑦= 𝑋′𝑋 A5𝑋V 𝑋𝛽+𝜀 =𝛽+ 𝑋′𝑋 A5𝑋V𝜀
𝐸 𝛽U
− 𝛽U
𝛽U u ^ _ − 𝛽 = 𝑋 ′ 𝑋 A 5 𝑋 V 𝜀
𝛽U −𝛽U = 𝑋′ΩA5𝑋A5𝑋VΩA5−𝑋′𝑋A5𝑋V𝜀
= 0 since both estimators are unbiased.
Lecture 16 GR5411 by Seyhan Erden 29
]^_ ]^_
u^_ u^_
11/11/20
The covariance 𝐶𝑜𝑣 𝛽Uqrs, 𝛽Uqrs − 𝛽Utrs , of the GLS estimator 𝛽Uqrs and the OLS estimator?
Therefore, U U U
𝐶𝑜𝑣 𝛽qrs, 𝛽qrs −𝛽rs |𝑋
= 𝐸 𝑋′ΩA5𝑋 A5𝑋VΩA5𝜀
= 𝐸 𝑋′ΩA5𝑋 A5𝑋VΩA5𝜀𝜀V
U U U
= 𝐸 𝛽qrs − 𝛽 𝛽qrs−𝛽trs ′ |𝑋
𝑋VΩA5 𝑋 A5𝑋VΩA5 − 𝑋V𝑋 A5𝑋V 𝜀 V|𝑋 𝑋V ΩA5𝑋 A5𝑋VΩA5 − 𝑋V𝑋 A5𝑋V V|𝑋
= 𝑋′ΩA5𝑋 A5𝑋VΩA5𝐸 𝜀𝜀V|𝑋 ΩA5𝑋 𝑋′ΩA5𝑋 A5 − 𝑋 𝑋′𝑋 A5
= 𝑋′ΩA5𝑋 A5𝑋VΩA5 𝜎2Ω ΩA5𝑋 𝑋′ΩA5𝑋 A5 − 𝑋 𝑋′𝑋 A5
= 𝜎2 𝑋′ΩA5𝑋 A5𝑋VΩA5Ω ΩA5𝑋 𝑋′ΩA5𝑋 A5 − 𝑋 𝑋′𝑋 A5
= 𝜎2 𝑋′ΩA5𝑋 A5𝑋V ΩA5𝑋 𝑋′ΩA5𝑋 A5 − 𝑋VΩA5 𝑋 A5𝑋V𝑋 𝑋V𝑋 A5
=0
Note that if 𝐸 𝜀𝜀V|𝑋 is defined as Ω, there will not be a 𝜎2 in this derivation. 11/11/20 Lecture 16 GR5411 by Seyhan Erden 30