Introduction to information system
Logistic Regression
Deema Abdal Hafeth
Bowei Chen
CMP3036M/CMP9063M Data Science
2016 – 2017
Objectives
Limitation of Linear Regression
Basic Concepts
Logit Function
Odds Ratio
Logistic Function
Logistic Regressions
Simple Logistic Regression
Multiple Logistic Regression
References
• James, G., Witten, D., Hastie, T., and Tibshirani, R. (2013). An introduction
to statistical learning. Springer. (Chapter 4)
• Abu-Mostafa, Yaser S., Malik Magdon-Ismail, and Hsuan-Tien Lin. Learning
from data. AMLBook, 2012. (Chapter 3)
• 𝑦 – Response variable
• 𝑥 – Predictor variable(s)
• 𝛽 – Intercept and slop
• 𝜀 – Error term
Linear
Regression
Simple
𝑦 = 𝑓 𝑥 = 𝛽0 + 𝛽1𝑥 + 𝜀
Multiple
𝑦 = 𝛽0 + 𝛽1𝑥𝑖,1 + 𝛽2𝑥𝑖,2 + …+ 𝛽𝑝𝑥𝑖,𝑝 + 𝜀
Price House size
1 420 5850
2 385 4000
3 495 3060
4 605 6650
5 610 6360
6 660 4160
7 660 3880
8 690 4160
9 838 4800
10 885 5500
… … …
Dataset
For this new house with size 4050 (sq ft), can we predict what is it the rent price?
Simple Linear Regression
Price House size
1 420 5850
2 385 4000
3 495 3060
4 605 6650
5 610 6360
6 660 4160
7 660 3880
8 690 4160
9 838 4800
10 885 5500
…
Response variable
Independent variable (x): Ppredictors
variable, feature or explanatory variable
Predictor
4050
𝑦 ≔ 𝑓 𝑥 = 𝛽0 + 𝛽1𝑥 + ε
Slop Intercept
𝑥
𝑦
𝑦
For this new house with size 4050 (sq ft), 4 bedrooms and 2 bathrooms,
can we predict what is it the rent price?
There Are Other Features of Houses
Price House size Bedrooms Bathrms Stories Driveway Recroom Fullbase
1 420 5850 3 1 2 1 0 1
2 385 4000 2 1 1 1 0 0
3 495 3060 3 1 1 1 0 0
4 605 6650 3 1 2 1 1 0
5 610 6360 2 1 1 1 0 0
6 660 4160 3 1 1 1 1 1
7 660 3880 3 2 2 1 0 1
8 690 4160 3 1 3 1 0 0
9 838 4800 3 1 1 1 1 1
10 885 5500 3 2 4 1 1 0
… … … … … … … … …
Multiple Linear Regression
Simple expression:
𝑦𝑖 = 𝛽0 + 𝛽1𝑥𝑖,1 + 𝛽2𝑥𝑖,2 + ⋯+ 𝛽𝑝𝑥𝑖,𝑝 + 𝜀𝑖,
Matrix expression:
𝒚 = 𝒙𝜷 + 𝜺,
where
𝐲 =
𝑦1
⋮
𝑦𝑛
, 𝜷 =
𝛽0
𝛽1
⋮
𝛽𝑝
, 𝒙 =
1 𝑥1,1 ⋯ 𝑥1,𝑝
⋮ ⋮ ⋱ ⋮
1 𝑥𝑛,1 ⋯ 𝑥𝑛,𝑝
, 𝜺 =
𝜀1
⋮
𝜀𝑛
Limitation of Linear Regression
Price Fullbase
1 420 1
2 385 0
3 495 0
4 605 0
5 610 0
6 660 1
7 660 1
8 690 0
9 838 1
10 885 0
… … …
Housing dataset
Response
variable
Predictor
Logit Function and Odds Ratio
The logit function of 𝑝, where 𝑝 is between 0 and 1, can be expressed as
logit 𝑝 = log
𝑝
1 − 𝑝
= log 𝑝 − log 1 − p
𝑝
1−𝑝
is called odds ratio
If 𝑝 = 0, logit 𝑝 → −∞
If 𝑝 = 1, logit 𝑝 → ∞
Logistic Function
The logit function is the inverse of logistic function. If we let 𝛼 = logit 𝑝 , then
logistic 𝛼 = logit−1 𝑝 =
1
e−𝛼 + 1
=
𝑒𝛼
1 + 𝑒𝛼
𝑒𝛼
1 + 𝑒𝛼
log
𝑝
1 − 𝑝
Simple Logistic Regression
The logit of the underlying probability 𝑝𝑖 is a linear function of the predictors
logit 𝑝𝑖 = 𝛽0 + 𝛽1𝑥𝑖 ,
then
𝑝𝑖 =
1
1 + 𝑒−(𝛽0+𝛽1𝑥𝑖)
=
𝑒𝛽0+𝛽1𝑥𝑖
1 + 𝑒𝛽0+𝛽1𝑥𝑖
.
We should predict yi = 1 when 𝑝𝑖 ≥ 0.5 and y𝑖 = 0 when 𝑝𝑖 < 0.5. This means guessing 1 whenever 𝛽0 + 𝛽1𝑥𝑖 is non-negative, and 0 otherwise. So logistic regression gives us a linear classifier. The decision boundary separating the two predicted classes is the solution of 𝛽0 + 𝛽1𝑥𝑖 = 0. Simple Logistic Regression Parameters Estimation We use maximum likelihood estimation (MLE) method to estimate parameters’ values. Simple, the likelihood function can be written as ℒ 𝛽0 , 𝛽1 = ℙ(𝒚 ∣ 𝒙, 𝛽0, 𝛽1) = 𝑝𝑖 𝑦𝑖 𝑛 𝑖=1 (1 − 𝑝𝑖) 1−𝑦𝑖 Then 𝛽0 , 𝛽1 = argmax log{ℒ 𝛽0 , 𝛽1 } We can quickly obtain 𝛽0, 𝛽1 in R for logistic regression. The algorithmic solution of parameters estimation is not required but I hope you could search it after class. 𝑛 observations 𝑦𝑖 = 0 or 1 𝑒𝛽0+𝛽1𝑥𝑖 1 + 𝑒𝛽0+𝛽1𝑥𝑖 bodysize survive 1 1.747326 0 2 1.752239 0 3 1.806183 0 4 1.812122 0 5 1.816628 0 6 1.847038 1 7 1.860676 0 8 1.897639 1 9 1.902124 0 10 1.904108 0 … … … Multiple Logistic Regression The logit of the underlying probability 𝑝𝑖 is a linear function of the predictors logit 𝑝𝑖 = 𝒙𝒊 𝑻𝜷, where 𝜷 = 𝛽0 𝛽1 ⋮ 𝛽𝑝 , 𝒙 = 1 𝑥1,1 ⋯ 𝑥1,𝑝 ⋮ ⋮ ⋱ ⋮ 1 𝑥𝑛,1 ⋯ 𝑥𝑛,𝑝 . Then 𝑝𝑖 = 𝑒𝒙𝒊 𝑻𝜷 1 + 𝑒𝒙𝒊 𝑻𝜷 . 𝑥𝑛 bodysize survive age 1 1.773196 0 34 2 1.775586 0 64 3 1.826180 0 30 4 1.832622 0 11 5 1.838073 0 22 6 1.850233 1 68 7 1.850588 0 13 8 1.865037 1 48 9 1.883509 0 22 10 1.886488 0 20 … … … … Summary Limitation of Linear Regression Basic Concepts Logit Function Odds Ratio Logistic Function Logistic Regressions Simple Logistic Regression Multiple Logistic Regression Thank You bchen@Lincoln.ac.uk mailto:bchen@Lincoln.ac.uk