ECON 61001: Review of OLS in simple linear regression
Alastair R. Hall
The University of Manchester
Alastair R. Hall
ECON61001: OLS Review 1 / 12
Outline of this revision session
Origins of OLS
Simple linear regression model
Intuition behind OLS
Formal definition of OLS
Other approaches and advantages of OLS
Resources: any introductory econometrics text such as Wooldridge (2019) Introductory Econometrics.
Alastair R. Hall
ECON61001: OLS Review 2 / 12
Origins of OLS
The origin of Ordinary Least Squares (OLS) is shrouded in controversy.
The method is published by
Adrien Legendre, a French mathematician, in 1805. Robert Adrain, an American, in 1808.
Carl Friedrich Gauss, a German, in 1809
but Gauss claimed he had been using the method since 1795.
Stigler (1981): “The method of least squares is the automobile of modern statistical analysis” but who was the “Henry Ford of statistics”?
Alastair R. Hall
ECON61001: OLS Review 3 / 12
Origins of OLS
Gauss was a very eminent mathematician whose key contributions include:
the Normal (or “Gaussian”) distribution. development of Least Squares theory Gaussian elimination
Alastair R. Hall
ECON61001: OLS Review 4 / 12
Origins of OLS
Stigler (1981) evaluates Gauss’s claim using historical records and concludes
Just as the automobile was not the product of one man of genius, so too the method of least squares is due to […] at least two independent discoverers. Gauss may well have been the first of these, but he was no Henry Ford of statistics. If there was any single scientist who first put the method within the reach of the common man, it was Legendre.(Stigler, 1981,p.472)
Alastair R. Hall
ECON61001: OLS Review 5 / 12
Simple linear regression
Suppose we are interesting in modeling the relationship between y, the annual salary of the CEO of a firm i, and xi the average return on equity for the CEO’s firm for the previous 3 years.
Assume simple linear regression model
yi = β0,1 +β0,2xi +ui = xi′β0 +ui
where
ui is the unobserved error term;
xi′ =(1,xi);
β0,1
β0 = β are unknown (regression) parameters. 0,2
Alastair R. Hall
ECON61001: OLS Review 6 / 12
Simple linear regression
Under assumptions discussed in the lecture, we have: E[yi |xi] = β0,1 + β0,2xi
So E[yi |xi] is linear function of xi but weights (the regression parameters) are unknown.
So collect sample on {yi , xi } and use this to estimate β0.
Alastair R. Hall
ECON61001: OLS Review 7 / 12
Simple linear regression
Suppose our data consists of the following five observations on (yi,xi):
{(1095, 14.1), (1001, 10.9), (1122, 23.5), (578, 5.9), (1145, 20.0)}.
Alastair R. Hall
ECON61001: OLS Review 8 / 12
Simple linear regression
No single line passes through all the points.
Choose line that comes “closest” to fitting the scatter plot → issue of how to measure distance of actual y from value predicted by line, β1 + β2x.
Alastair R. Hall
ECON61001: OLS Review 9 / 12
Simple linear regression and OLS
Define ui (β) = yi − β1 − β2xi
Measure of distance from line must be non-negative: in OLS we measure this distance by {ui (β)}2
OLS estimator of (β1, β2) is value that minimizes 5i=1{ui (β)}2.
Alastair R. Hall
ECON61001: OLS Review 10 / 12
OLS in the simple linear regression model
In this example the OLS line is:
yˆi = 573.63 + 27.86xi
Alastair R. Hall
ECON61001: OLS Review 11 / 12
OLS in the simple linear regression model
Notice that {ui(β)}2 is just one possible measure of distance.
For example, could also use ui (β):
If choose (β1, β2) to minimize 5 ui (β) → Least Absolute i=1
Deviation (LAD) regression
LAD actually proposed in 1757 by Roger Boscovich but OLS became the “automobile of modern statistical analysis” because:
calculus of OLS is far easier
OLS can be shown to have some desirable statistical properties (see lectures).
Alastair R. Hall
ECON61001: OLS Review 12 / 12