Linear regression
Regression problem
Regression problems occur often in machine learning, for example, if you want to estimate
weight, age or height of a person in an image. In regression problems you need to estimate
some continuous variable, but by “continuous” we only mean characteristics of the problem.
Typically the continous value is a real number, , but this is not necessarily the case.
Think, for example, estimation of pose of an object in 3D world.
In the case of age estimation from an image, the ML problem is defined similar to what we
know from the first lecture, , where is the image of a human face, is age (e.g.
) and is the ML model that maps image to the integer/real number of age.
Example 2.1 Let’s assume you are a medical hobit who has money to buy only one new
instrument, a scale for measuring weight or a measure to measure height. You want to save
some money and as a ML enthusiast you think that you can estimate the weight if you buy
only the cheap measure for height. You simply measure height and use some ML model to
estimate height .
Linear model
Our model is a line and the equation should be familiar from high school:
Fitting a line
Fitting a line essentially means that the model parameters and are tuned to fit your data.
y ∈ R
y = f(x) x y
∈ [0, 130] f
y
x
y = f(x)
y = ax + b . (1)
a b
lec02_linear_regression http://localhost:8888/nbconvert/html/lec02_linear_regr…
1 of 7 9/1/21, 16:03
Can you think of some methods immediately?
You might remember that is called slope and is called intersept, and for obvious reasons.
Zero points
In [2]:
import matplotlib.pyplot as plt
import numpy as np
x = np.linspace(0,2.0,10)
#
# Plot some pre-defined lines
plt.clf()
a=0
b=0
y = a*x+b
plt.plot(x,y,label=’a=0 b=0′)
a=10
b=0
y = a*x+b
plt.plot(x,y,label=’a=10 b=0′)
a=0
b=40
y = a*x+b
plt.plot(x,y,label=’a=0 b=40′)
a=10
b=40
y = a*x+b
plt.plot(x,y,label=’a=10 b=40′)
a=0
b=80
y = a*x+b
plt.plot(x,y,label=’a=0 b=80′)
plt.legend()
plt.title(‘Lines y=ax+b with selected values of a and b (x: height, y: weight)’
plt.xlabel(‘height [m]’)
plt.ylabel(‘Weight [kg]’)
plt.axis([0,2,0,150])
plt.show()
a b
lec02_linear_regression http://localhost:8888/nbconvert/html/lec02_linear_regr…
2 of 7 9/1/21, 16:03
One point
We measure one hobit:
In [3]:
# Plot random lines
for foo in range(10):
a = np.random.uniform(-50.0,+50.0,1)
b = np.random.uniform(0.0,+150.0,1)
x = np.linspace(0,2.0,10)
y = a*x+b
plt.plot(x,y)
plt.title(‘Lines y=ax+b with random values of a and b (x: height, y: weight)’
plt.xlabel(‘height [m]’)
plt.ylabel(‘Weight [kg]’)
plt.axis([0,2,0,150])
plt.show()
(x1, y1) = (1.11, 85.0)
lec02_linear_regression http://localhost:8888/nbconvert/html/lec02_linear_regr…
3 of 7 9/1/21, 16:03
Two points
In [4]:
# Plot lines through one given point
plt.clf()
# point 1
x1 = 1.11
y1 = 85.0
plt.plot(x1, y1,’bo’)
plt.annotate(“(1.11,85.0)”, (x1, y1), textcoords=”offset points”,xytext=(-0.1
a=0
b=y1-a*1.11
y = a*x+b
plt.plot(x,y,label=’a=0′)
a=1
b=y1-a*1.11
y = a*x+b
plt.plot(x,y,label=’a=1′)
a=5
b=y1-a*1.11
y = a*x+b
plt.plot(x,y,label=’a=5′)
a=10
b=y1-a*1.11
y = a*x+b
plt.plot(x,y,label=’a=10′)
a=20
b=y1-a*1.11
y = a*x+b
plt.plot(x,y,label=’a=20′)
a=40
b=y1-a*1.11
x = np.linspace(0,2.0,10)
y = a*x+b
plt.plot(x,y,label=’a=40′)
plt.legend()
plt.title(‘Lines that follow 85=a*1.11+b’)
plt.xlabel(‘height [m]’)
plt.ylabel(‘weight [kg]’)
plt.axis([0,2,0,150])
plt.show()
lec02_linear_regression http://localhost:8888/nbconvert/html/lec02_linear_regr…
4 of 7 9/1/21, 16:03
N points
In [5]:
# Plot the line through two given points
plt.clf()
# point 1
x1 = 1.11
y1 = 85.0
plt.plot(x1, y1,’bo’)
plt.annotate(“(1.11,85.0)”, (x1, y1), textcoords=”offset points”,xytext=(-0.1
x2 = 1.52
y2 = 110.0
plt.plot(x2, y2,’bo’)
plt.annotate(“(1.52,110.0)”, (x2, y2), textcoords=”offset points”,xytext=(-
a=60.98
b=17.32
x = np.linspace(0,2.0,10)
y = a*x+b
plt.plot(x,y,’b-‘,label=’y=60.98x+17.32’)
plt.legend()
plt.title(‘Line through two known points’)
plt.xlabel(‘Height [m]’)
plt.ylabel(‘Weight [kg]’)
plt.axis([0,2,0,150])
plt.show()
In [6]:
# N points
np.random.seed(42) # to always get the same points
N = 50
x = np.random.normal(1.1,0.3,N)
a_gt = 50.0
b_gt = 20.0
y_noise = np.random.normal(0,8,N) # Measurements from the class 1\n”,
y = a_gt*x+b_gt+y_noise
plt.plot(x,y,’bo’)
plt.title(‘Training data’)
plt.xlabel(‘Height [m]’)
plt.ylabel(‘Weight [kg]’)
plt.axis([0,2,0,150])
plt.show()
lec02_linear_regression http://localhost:8888/nbconvert/html/lec02_linear_regr…
5 of 7 9/1/21, 16:03
We cannot anymore fit a line that goes through every point so instead we define an error
measure we will minimize.
Definition 2.2.: Mean Squared Error (MSE)
The Mean Squared Error for estimates and known (ground truth) values is computed
from
ŷ y
ϵMSE =
N
∑
i=1
(yi − ŷ i)
2 (2)
1
N
In [7]:
from matplotlib.colors import LogNorm
# Compute MSE heat map for different a and b
MSE_ab = np.empty([11,11])
for ai,a in enumerate((range(0,110,10))):
for bi, b in enumerate((range(0,110,10))):
y_hat = a*x+b
MSE_ab[ai][bi] = np.sum((y-y_hat)**2)/N
plt.imshow(MSE_ab,extent= [-5,105,-5,105], cmap=’hot’, norm=LogNorm(), interpolation
plt.colorbar()
plt.ylabel(‘a’)
plt.xlabel(‘b’)
plt.title(‘MSE for different values of a and b’)
plt.show()
lec02_linear_regression http://localhost:8888/nbconvert/html/lec02_linear_regr…
6 of 7 9/1/21, 16:03
During lectures we start to derive the formulas for solving and for points and finishing
that and writing the code will be your next week exercise.
Line fitting in matrix form
We may conviniently define the inputs and outputs as vectors (denoted by bold symbols) and
matrices (denoted by capital bold symbols)
when the group of all linear equations that must hold is simply put as:
The MSE computation with matrices is also pretty straightforward:
Solving the weight vector of the above equation is simpler than without the matrix form and
actually that gives us a solution that is easy to extend. But let’s continue that the next time.
References
C.M. Bishop (2006): Pattern Recognition and Machine Learning, Chapter 3.
S.M. Sigler (1981): Gauss and the invention of least squares, The Annals of Statistics, Vol. 9,
No. 3.
a b N
y =
⎛⎜⎜⎜⎜⎝
y1
y2
⋮
yN
⎞⎟⎟⎟⎟⎠
, X =
⎛⎜⎜⎜⎜⎝
1 x1
1 x2
⋮
1 xN
⎞⎟⎟⎟⎟⎠
, ja w = ( b
a
) , (3)
y = Xw . (4)
ϵMSE =
N
∑
i=1
(yi − ŷ i)
2 = (y − ŷ)T (y − ŷ) = (y − Xw)T (y − Xw)
= (yT y − yT Xw − (Xw)T y + (Xw)T Xw)
= (yT y − 2yT Xw + wT XT Xw) .
(5)
1
N
1
N
1
N
1
N
1
N
w
lec02_linear_regression http://localhost:8888/nbconvert/html/lec02_linear_regr…
7 of 7 9/1/21, 16:03