ECONOMETRICS I ECON GR5411
Lecture 9 – Measures of Fit in matrix form. Asymptotic Properties of OLS estimator
by
Seyhan Erden Columbia University MA in Economics
Measures of Fit
Coefficient of Determination:
𝑅” = 𝐸𝑥𝑝𝑙𝑎𝑖𝑛𝑒𝑑 𝑠𝑢𝑚 𝑜𝑓 𝑠𝑞𝑢𝑎𝑟𝑒𝑠 𝑇𝑜𝑡𝑎𝑙 𝑠𝑢𝑚 𝑜𝑓 𝑠𝑞𝑢𝑎𝑟𝑒𝑠
𝑅” = 1 − 𝑆𝑢𝑚 𝑜𝑓 𝑠𝑞𝑢𝑎𝑟𝑒𝑠 𝑟𝑒𝑠𝑖𝑑𝑢𝑎𝑙𝑠 𝑇𝑜𝑡𝑎𝑙 𝑠𝑢𝑚 𝑜𝑓 𝑠𝑞𝑢𝑎𝑟𝑒𝑠
𝑅”=𝛽9;𝑋;𝑀>𝑋𝛽9=1− 𝜀̂;𝜀̂ =1− 𝑦;𝑀𝑦
𝑦;𝑀>𝑦 𝑦;𝑀>𝑦 𝑦; 𝑀>𝑦
𝑅” measures the proportion of variation in 𝑦 explained by variation in
regressors.
In class: showing ∑ 𝜀̂ − 𝜀̅ ” = ∑ 𝜀̂” , ∑ 𝜀̂ − 𝜀̅ ” = ∑ 𝜀̂” − 𝜀̅ 2 ∑ 𝜀̂ −
0 ≤ 𝑅” ≤ 1
”
EEEEE
𝑛𝜀̅ but𝜀̅=∑𝜀̂⁄𝑛=0because∑𝜀̂ =0. EE
10/12/20 Lecture 8 GR5411 by Seyhan Erden 2
Measures of Fit
Then,
𝑦;𝑀>𝑦 = 𝛽9;𝑋;𝑀>𝑋𝛽9 + 𝜀̂;𝜀̂
Notation differences in textbooks
𝑆𝑆𝑇 = 𝑆𝑆𝑅 + 𝑆𝑆𝐸 Greene
𝑇𝑆𝑆 = 𝐸𝑆𝑆 + 𝑆𝑆𝑅 (𝑜𝑡h𝑒𝑟 𝑡𝑒𝑥𝑡𝑏𝑜𝑜𝑘𝑠)
10/12/20 Lecture 8 GR5411 by Seyhan Erden 3
Adjusted 𝑅”:
Ø𝑅” can be negative if regression has no constant.
Ø𝑅” will never decrease when a regressor is added, favoring large models with irrelevant regressors.
ØAdjusted 𝑅” is given as 1 𝑅S”=1− 𝑛−𝑘
𝜀̂;𝜀̂
1 𝑦;𝑀>𝑦 𝑛−1
ØThe connection between 𝑅” and 𝑅S” is 𝑅S ” = 1 − 𝑛 − 1 1 − 𝑅 ”
𝑛−𝑘
10/12/20 Lecture 8 GR5411 by Seyhan Erden 4
Measures of Fit ” Another expression for 𝑅
is
∑ 𝜀 ̂ ”
𝜎T ”
𝑅” = 1 − E
where 𝜎T” = 𝑛WX ∑ 𝑦 − 𝑦S ” and 𝜎T” = 𝑛WX ∑ 𝜀̂”. “VEE
𝑅 can be viewed as an estimator of the population parameter 𝜌”:
𝜌”=𝑉𝑎𝑟𝒙;E𝛽 =1−𝜎” 𝑉𝑎𝑟 𝑦E 𝜎V”
= 1 − EV
∑ 𝑦 − 𝑦S ”
𝜎T ”
10/12/20 Lecture 8 GR5411 by Seyhan Erden
5
Measures of Fit
However, 𝜎T” and 𝜎T” are biased estimators. We V”
must replace these with unbiased versions 𝑠 and 𝜎\” = 𝑛−1 WX∑ 𝑦 −𝑦S “.Thisgivesusthe
V E 𝟐 _𝟐 formula for adjusted 𝑹 , 𝑹
𝑠”
𝑛−1 ∑𝜀̂” E
𝑛 − 𝑘 ∑ 𝑦 E − 𝑦S ”
𝑅S”=1− =1−
𝜎\ ” V
10/12/20
Lecture 8 GR5411 by Seyhan Erden 6
Now Asymptotic Distribution of the OLS Estimator, 𝛽9:
If the sample size is large and the first four assumptions are satisfied, then the OLS estimator has an asymptotic joint normal distribution, the heteroskedasticity-robust estimator of the covariance is consistent, and the heteroskedasticity- robust OLS t-statistic has an asymptotic standard normal distribution (from Lecture 2 slides) and a multivariate extension of the central limit theorem.
10/12/20 Lecture 8 GR5411 by Seyhan Erden 7
Asymptotic Normality of 𝛽9:
In large samples, the OLS estimator has the
multivariate normal asymptotic distribution
𝑛𝛽9−𝛽 →a 𝑁0,Σ where Σ e gfWg = 𝑄WXΣi𝑄WX
where 𝑄 is 𝑘×𝑘 matrix of second moments of the regressor–thatis𝑄=𝐸 𝑋E𝑋E; –andΣi is𝑘×𝑘 covariance matrix of 𝑉E = 𝑋E𝜀E – that is, Σi =
𝐸 𝑉E𝑉E; . Note that 𝑉E’s are i.i.d.
e gf W g
10/12/20 Lecture 8 GR5411 by Seyhan Erden 8
Asymptotic Normality of 𝛽9: Derivation of
𝑛𝛽9−𝛽 →a 𝑁0,Σ
can be shown as:
e gf W g
Then,
𝛽9 = 𝑋 ; 𝑋 W X 𝑋 ; 𝑦
= 𝑋;𝑋WX𝑋; 𝑋𝛽+𝜀 =𝛽+ 𝑋;𝑋WX𝑋;𝜀
𝛽9 − 𝛽 = 𝑋 ; 𝑋 W X 𝑋 ; 𝜀
10/12/20
Lecture 8 GR5411 by Seyhan Erden 9
Asymptotic Normality of 𝛽9:
Then,
𝑋;𝑋 WX 𝑋;𝜀 𝑛𝑛
𝑋;𝑋 WX 𝑋;𝜀 𝑛𝑛𝑛
𝑋;𝑋 WX 𝑋;𝜀
𝑛𝑛
or,
𝛽9 − 𝛽 =
or,
𝛽9 − 𝛽 =
𝑛 𝛽9 − 𝛽 =
10/12/20
Lecture 8 GR5411 by Seyhan Erden 10
Asymptotic Normality of 𝛽9:
𝑋;𝑋 WX 𝑋;𝜀 𝑛𝑛
We argue that the denominator matrix is consistent
for 𝑄k, that is
𝑋 ; 𝑋 →m 𝑄
𝑛
and the numerator matrix ; 𝑋𝜀
𝑛
obeys the multivariate Central Limit Theorem
𝑛 𝛽9 − 𝛽 =
10/12/20 Lecture 8 GR5411 by Seyhan Erden 11
Heteroskedasticity – Robust Standard Errors:
The heteroskedasticity – robust standard errors of Σ e gfWg is obtained by replacing the population
moments in its definition by sample moments. Accordingly, the heteroskedasticity – robust estimator
of the covariance matrix of 𝑛 𝛽9 − 𝛽 is 𝑋;𝑋 WX 𝑋;𝑋 WX
where
n1e
ΣnegfWg= 𝑛 Σnif 𝑛
Σf = o𝑋𝑋;𝜀̂” i𝑛−𝑘EEE
10/12/20
Lecture 8 GR5411 by Seyhan Erden 12
EpX
Asymptotics in more detail:
Large sample properties of LS estimator:
Two crucial assumptions
1. A5a. (modification of A5) 𝒙E,𝜀E ,𝑖=1,….,𝑛
is a sequence of independently, identically distributed observations
2. The second concerns the behavior of the data in large samples 𝑋;𝑋
plim 𝑛 =𝑄 e→w
10/12/20 Lecture 8 GR5411 by Seyhan Erden
13
Asymptotics in more detail:
Also it is important to note that all necessary moments are finite
𝐸𝜀Ex <∞,𝐸 𝑥E x <∞ and
𝐸 𝑥E𝑥E; ≡ 𝑄 exists and is positive definite. 𝐸 𝜀E|𝑥E =0 ⇒ 𝐸 𝑥E𝜀E =0 butnotthe
other way around
10/12/20 Lecture 8 GR5411 by Seyhan Erden
14
Asymptotics:
𝑄 is a positive definite matrix
Note that we can write the LS estimator as
𝛽9 = 𝑋 ; 𝑋 W X 𝑋 ; 𝑦
= 𝑋;𝑋WX𝑋; 𝑋𝛽+𝜀 =𝛽+ 𝑋;𝑋WX𝑋;𝜀.
𝑋;𝑋 WX 𝑋′𝜀
𝑛𝑛
1e WX1e
=𝛽+ 𝑛o𝑥E𝑥E; 𝑛o𝑥E𝜀E EpX EpX
=𝛽+
10/12/20
Lecture 9 GR5411 by Seyhan Erden
15
Consistency:
To show that 𝛽9 is consistent we need to show 𝛽9 →m 𝛽
Same as 𝑝𝑙𝑖𝑚𝛽9 = 𝛽.
Since according to assumption A5.a 𝒙E , 𝜀E are i.i.d. across all observations 𝑖 = 1, ... 𝑛, and since
𝐸 𝑥E𝑥E;
the term
≡ 𝑄 then by weak LLN we can write n1em
𝑄≡𝑛o𝑥E𝑥E; →𝑄 EpX
10/12/20
Lecture 9 GR5411 by Seyhan Erden
16
Consistency:
and since 𝑄 exists and inverse is a linear operation, we can write
n 1e WXm 𝑄WX= 𝑛o𝑥E𝑥E; →𝑄WX
EpX
Also, by the same assumptions and weak LLN the 2nd term converges to 0:
1em
𝑛o𝑥E𝜀E →𝐸𝑥E𝜀E =0
EpX
10/12/20
Lecture 9 GR5411 by Seyhan Erden
17
Consistency:
Hence,
9 1e WX1e
𝑝𝑙𝑖𝑚𝛽=𝛽+𝑝𝑙𝑖𝑚 𝑛o𝑥E𝑥E; 𝑛o𝑥E𝜀E EpX EpX
= 𝛽 + 𝑄WX 0 =𝛽
We proved that under the assumptions given above
𝛽9 is consistent:
𝛽9 →m 𝛽
10/12/20 Lecture 9 GR5411 by Seyhan Erden
18
Asymptotic normality:
Recall that
9 1e WX1e
𝛽−𝛽= 𝑛o𝑥E𝑥E; 𝑛o𝑥E𝜀E EpX EpX
Pre-multiplying both sides by 𝑛
9 1e WX 1e
𝑛𝛽−𝛽 = 𝑛o𝑥E𝑥E; 𝑛 𝑛o𝑥E𝜀E EpX EpX
Some facts about this equation...next slide
10/12/20 Lecture 9 GR5411 by Seyhan Erden
19
Asymptotic normality:
Some facts about this equation:
1. The first term is: WX
1e
𝑝𝑙𝑖𝑚𝑛o𝑥E𝑥E; =𝑄WX=𝐸𝑥E𝑥E; WX
EpX
2. Let 𝑤E = 𝑥E𝜀E, then 𝐸 𝑤E = 𝐸 𝑥E𝜀E = 0, then
the 2nd term is:
1ee
𝑛𝑛o𝑥E𝜀E =𝑛o𝑤E−𝐸𝑤E EpX EpX
= 𝑛 𝑤_ − 𝐸 𝑤 E
10/12/20
Lecture 9 GR5411 by Seyhan Erden
20
Since 𝑤E is iid, by CLT, we can write 𝑛𝑤_−𝐸𝑤E →a 𝑁0,𝑣𝑎𝑟𝑤E
where
𝑣𝑎𝑟 𝑤E =𝐸 𝑤E𝑤E; =𝐸 𝑥E𝑥E;𝜀E" ≡Ω
Combining 1 and 2 above, we can write
9 1e WX 1e
𝑛𝛽−𝛽 = 𝑛o𝑥E𝑥E; EpX
→a 𝑄WX𝑁0,Ω →a 𝑁 0,𝑄WXΩ𝑄WX
𝑛 𝑛o𝑥E𝜀E EpX
10/12/20
Lecture 9 GR5411 by Seyhan Erden
21
Asymptotic normality:
We proved that
or
where and
𝑛𝛽9−𝛽→a 𝑁0,𝑉g 𝛽9 →a 𝑁 0,𝑛1𝑉g
𝑉g = 𝑄WXΩ𝑄WX
Ω = 𝐸 𝑥 E 𝑥 E; 𝜀 E" 𝑄WX=𝐸𝑥E𝑥E; WX
10/12/20
Lecture 9 GR5411 by Seyhan Erden
22
Under Heteroskedasticity:
𝑎𝑠𝑦𝑚. 𝑣𝑎𝑟 𝛽9 = 𝑉g = 𝑄WXΩ 𝑄WX
where
and can be consistently estimated by
f1em Ω≡ o𝑥𝑥;𝜀̂" →Ω
Ω = 𝐸 𝑥 E 𝑥 E; 𝜀 E"
EEE
𝑛EpX
and since 𝑄WX = 𝐸 𝑥E𝑥E; WX is consistently estimated by
n 1e WX 1 𝑄WX= 𝑛o𝑥E𝑥E; = 𝑛𝑋;𝑋
EpX
So, under heteroskedasticity asymptotic variance of 𝛽 can
be estimated as 𝑄nWXΩf 𝑄nWX
10/12/20 Lecture 9 GR5411 by Seyhan Erden
9
23
Asymptotic Variance Under Homoskedasticity:
Show: 𝑎𝑠𝑦𝑚. 𝑣𝑎𝑟 𝛽9 = 𝑉g = 𝑄WXΩ 𝑄WX = 𝜎"𝑄WX Ω can be simplified by L.I.E.
Ω = 𝐸 𝑥 E 𝑥 E; 𝜀 E"
= 𝐸 𝐸 𝑥 E 𝑥 E; 𝜀 E" | 𝑥 E
= 𝐸 𝑥 E 𝑥 E; 𝐸 𝜀 E" | 𝑥 E
= 𝜎 " 𝐸 𝑥 E 𝑥 E; = 𝜎"𝑄
Thus,
𝑉g = 𝑄WXΩ 𝑄WX = 𝑄WX𝜎"𝑄𝑄WX = 𝜎"𝑄WX
10/12/20 Lecture 9 GR5411 by Seyhan Erden
24
Under Homoskedasticity:
If 𝑠" is consistent (i.e. 𝑠" →m 𝜎") and 𝑄WX = 𝐸 𝑥E𝑥E; WX is consistently estimated by
n 1e WX 1 𝑄WX= 𝑛o𝑥E𝑥E; = 𝑛𝑋;𝑋
EpX
So, under homoskedasticity asymptotic variance of 𝛽9 can be estimated as 𝑠"𝑄nWX
10/12/20 Lecture 9 GR5411 by Seyhan Erden
25
Consistency of 𝑠": (already showed 𝑠" is unbiased)
To complete the derivation of the asymptotic properties of 𝛽f we need an estimator of Let 𝑉g =
𝑎𝑠𝑦𝑚.𝑣𝑎𝑟𝛽9 =ÉÑ𝑄WX e
The purpose here is to assess the consistency of 𝑠" as an estimator of 𝜎".
Expanding
produces
𝑠" = 1 𝜀;𝑀𝜀 𝑛−𝑘
10/12/20
Lecture 9 GR5411 by Seyhan Erden
26
Consistency of 𝑠":
Expanding
𝑠" = 1 𝜀;𝑀𝜀 𝑛−𝑘
produces 1
𝑠" = 𝑛 − 𝑘 𝜀; 𝐼 − 𝑋 𝑋;𝑋 WX𝑋′ 𝜀
= 1 𝜀;𝜀 − 𝜀′𝑋 𝑋;𝑋 WX𝑋′𝜀 𝑛−𝑘
= 𝑛 𝜀;𝜀− 𝜀′𝑋 𝑋;𝑋 WX 𝑋′𝜀 𝑛−𝑘𝑛𝑛𝑛𝑛
10/12/20
Lecture 9 GR5411 by Seyhan Erden
27
Consistency of 𝑠":
To show
𝑋;𝑋 WX
𝑝𝑙𝑖𝑚 𝑠" 𝑛 = 𝜎"𝑄WX
𝑠"= 𝑛 𝜀;𝜀−𝜀′𝑋 𝑋;𝑋WX 𝑋′𝜀 𝑛−𝑘𝑛𝑛𝑛𝑛
ØThe 1st term in brackets converges in probability to 𝜎"
ØThe probability limit of the 2nd term is zero because 𝑝𝑙𝑖𝑚𝑤_ = 0
10/12/20 Lecture 9 GR5411 by Seyhan Erden
28
Consistency of 𝑠": Then,
9 𝜎" WX 𝑎𝑠𝑦𝑚.𝑣𝑎𝑟 𝛽 = 𝑛 𝑄
which can be estimated by
Ü9 " ; WX 𝑎𝑠𝑦𝑚.𝑣𝑎𝑟𝛽=𝑠 𝑋𝑋
10/12/20 Lecture 9 GR5411 by Seyhan Erden
29
Useful Notation:
𝑄 á á = 𝐸 𝑥 E 𝑥 E; , 𝑄 á V = 𝐸 𝑥 E 𝑦 E , 𝑄 á à = 𝐸 𝑥 E 𝜀 E
In the previous slides we used 𝑄 instead of 𝑄áá for simplicity because we did not need to distinguish it from others, but in future we will use 𝑄áá (for example under Instrumental Variables)
Also note that the corresponding sample moments
are
n1e1
𝑄 á á = 𝑛 o 𝑥 E 𝑥 E; = 𝑛 𝑋 ; 𝑋
EpX
10/12/20
Lecture 9 GR5411 by Seyhan Erden
30
Useful Notation:
𝑄áV =𝐸 𝑥E𝑦E , 𝑄áà =𝐸 𝑥E𝜀E
The corresponding sample moments
n1e1; 𝑄áV =𝑛o𝑥E𝑦E =𝑛𝑋𝑦
EpX
n1e1; 𝑄áà =𝑛o𝑥E𝜀E =𝑛𝑋𝜀
EpX
and
10/12/20
Lecture 9 GR5411 by Seyhan Erden
31