CS计算机代考程序代写 database flex Introduction to Machine Learning

Introduction to Machine Learning
Sarat C. Dass
Department of Mathematical and Computer Sciences Heriot-Watt University Malaysia Campus
Sarat C. Dass Department of Mathematical and ComInptruotdeurcStcioienntcoesMHaecrhioint-eWLaetatrnUingiversity Malaysia Campus 1/102

Introduction to Machine Learning
Sarat C. Dass Department of Mathematical and ComInptruotdeurcStcioienntcoesMHaecrhioint-eWLaetatrnUingiversity Malaysia Campus 2/102

What is Learning?
Let’s begin with a simple example. Suppose that you are a consultant hired to provide advice on how to improve sales of a particular product.
The Advertising data set consists of the sales of a product in 200 different markets, along with advertising budgets for the product in each of those markets for three different media: TV, radio, and newspaper.
The data are displayed as below:
0 50 100 150 200 250 300 0 10 20 30 40 50 0 20 40 60 80 100
TV Radio Newspaper
Sarat C. Dass Department of Mathematical and ComInptruotdeurcStcioienntcoesMHaecrhioint-eWLaetatrnUingiversity Malaysia Campus
3/102
Sales
5 10 15 20 25
Sales
5 10 15 20 25
Sales
5 10 15 20 25

What is Statistical Learning? (cont.)
What do you see from the plots? What do you conclude?
It is not possible for our client to directly increase sales of the product. On the other hand, they can control the advertising expenditure in each of the three media.
We determine if there is an association between advertising and sales, then we instruct our client to adjust advertising budgets, thereby indirectly increasing sales.
In other words, our goal is to develop an accurate model that can be used to predict sales on the basis of the three media budgets.
Sarat C. Dass Department of Mathematical and ComInptruotdeurcStcioienntcoesMHaecrhioint-eWLaetatrnUingiversity Malaysia Campus 4/102

Terms and Symbols Used for Statistical Learning
In this setting, the advertising budgets are input variables while sales is an output variable.
The inputs are typically denoted using the variable symbol X with subscripts to distinguish them, e.g., X1, X2, X3, etc.
So X1 might be the TV budget, X2 the radio budget, and X3 the newspaper budget.
The output variable is usually denoted by the symbol Y . Here, Y = sales.
The inputs go by different names, such as predictors, independent variables, features, or sometimes just variables.
The output variable is variable often called the response or dependent variable.
Throughout the lectures and the textbook, we will use all of these terms interchangeably.
Sarat C. Dass Department of Mathematical and ComInptruotdeurcStcioienntcoesMHaecrhioint-eWLaetatrnUingiversity Malaysia Campus
5/102

The Statistical Model
More generally, suppose that we observe a quantitative response Y and p different predictors, X1, X2, · · · , Xp.
We assume that there is some relationship between Y and
X = (X1,X2,··· ,Xp), which can be written in the very general form
Y =f(X)+ε
Here f (X ) is some fixed but unknown function of X and ε is a
random error term, which is independent of X and has mean zero. In this formulation, f (X ) represents the systematic information that
X provides about Y .
Let’s look at another example …
Sarat C. Dass Department of Mathematical and ComInptruotdeurcStcioienntcoesMHaecrhioint-eWLaetatrnUingiversity Malaysia Campus 6/102

Another example: Example 2
As another example, consider Y = income and
X = years of education for 30 individuals. The plot is given as below:
The plot suggests that one might be able to predict income using years of education.
However, the function Y = f (X ) that connects X to Y is gnerally unknown. In this situation, one must estimate f based on the observed points, call this fˆ(X).
To explain how this estimation will be performed, we use simulated datawherethetruef(X)≡E(Y|X)isknown. Thetruef(X)is shown by the blue curve in the right-hand panel of the figure on the next slide.
The vertical lines represent the error terms ε. We note that some of the 30 observations lie above the blue curve and some lie below it; overall, the errors have approximately mean zero.
Sarat C. Dass Department of Mathematical and ComInptruotdeurcStcioienntcoesMHaecrhioint-eWLaetatrnUingiversity Malaysia Campus 7/102

Figures for Example 2
Income
20 30 40 50 60 70 80
Income
20 30 40 50 60 70 80
10 12 14 16 18 20 22 10 12 14 16 18 20 22
Education Education
The vertical lines represent the error terms ε. We note that some of the 30 observations lie above the blue curve and some lie below it; overall, the errors have approximately mean zero.
Sarat C. Dass Department of Mathematical and ComInptruotdeurcStcioienntcoesMHaecrhioint-eWLaetatrnUingiversity Malaysia Campus 8/102

More general f s
In general, the function f (X ) may involve more than one input variable.
In Figure 2.3 of the textbook, we plot income as a function of years of education and seniority: f (X ) is a 2D surface that must be estimated based on the observed data. But errors have same characteristics.
Figure: This figure is Figure 2.3 taken from “An Introduction to Statistical Learning, with applications in R” (Springer, 2013) with permission from the authors: G. James, D. Witten, T. Hastie and R. Tibshirani
Sarat C. Dass Department of Mathematical and ComInptruotdeurcStcioienntcoesMHaecrhioint-eWLaetatrnUingiversity Malaysia Campus 9/102

The best regressor: formulation and derivation
Suppose we wish to predict Y by f (X ).
Let (X,Y) ∼ π(x,y) for which the marginal of X and the conditional
of Y given X are denoted by π(x) and π(y|x), respectively. The mean square error (MSE) is given by
MSE ≡Eπ(x,y)[Y −f(X)]2
=Eπ(x,y)[Y −E(Y|X)+E(Y|X)−f(X)]2
=Eπ(x,y)[Y −E(Y|X)]2 +Eπ(x,y)[E(Y|X)−f(X)]2 =Eπ(x)Eπ(y|x)[Y −E(Y|X)]2 +Eπ(x)[E(Y|X)−f(X)]2 = Eπ(x ) [Var(Y |X )] + Eπ(x ) [E (Y |X ) − f (X )]2
Thus,MSE isminimizedwhenf(X)=E(Y|X)whichisthe conditional expectation of Y given X calculated with respect to π(y|x).
E(Y|X) is the best regressor or best predictor for Y based on X.
Sarat C. Dass Department of Mathematical and ComInptruotdeurcStcioienntcoesMHaecrhioint-eWLaetatrnUingiversity Malaysia Campus
10/102

Steps of the proof
To get the third equality from the second, note that
Eπ(x,y) [Y −E(Y|X)+E(Y|X)−f(X)]2
=Eπ(x,y)􏰕(Y −E(Y|X))2 −2(Y −E(Y|X))(E(Y|X)−f(X))
+(E(Y|X)−f(X))2􏰖
=Eπ(x,y)(Y −E(Y|X))2 +Eπ(x,y)(E(Y|X)−f(X))2
since the term Eπ(x,y) [(Y −E(Y|X))(E(Y|X)−f(X))]
=Eπ(x)Eπ(y|x)[(Y −E(Y|X))(E(Y|X)−f(X))] =Eπ(x)􏰕(E(Y|X)−f(X))Eπ(y|x)(Y −E(Y|X))􏰖
(since E(Y|X)−f(X) is a function of X only)
= Eπ(x ) [(E (Y |X ) − f (X ))(E (Y |X ) − E (Y |X ))] = 0
Sarat C. Dass Department of Mathematical and ComInptruotdeurcStcioienntcoesMHaecrhioint-eWLaetatrnUingiversity Malaysia Campus
11/102

Steps of the proof (cont.)
To get the fourth and fifth equalities from the third, note that
Eπ(x,y)(Y −E(Y|X))2 +Eπ(x,y)(E(Y|X)−f(X))2 =Eπ(x)Eπ(y|x)(Y −E(Y|X))2 +Eπ(x)Eπ(y|x)(E(Y|X)−f(X))2 = Eπ(x)Var(Y|X)+Eπ(x)(E(Y|X)−f(X))2
since E(Y|X)−f(X) does not depend on Y; it depends on X only, and hence
Eπ(x)Eπ(y|x)(E(Y|X)−f(X))2
= Eπ(x) 􏰕(E(Y|X)−f(X))2 Eπ(y|x)(1)􏰖 = Eπ(x)(E(Y|X)−f(X))2,
andVar(Y|X)≡Eπ(y|x)(Y−E(Y|X))2 isthedefinitionofthe conditional variance of Y given X.
Sarat C. Dass Department of Mathematical and ComInptruotdeurcStcioienntcoesMHaecrhioint-eWLaetatrnUingiversity Malaysia Campus
12/102

The learning target
Recall the decomposition of the MSE as
MSE =Eπ(x)[Var(Y|X)]+Eπ(x)[E(Y|X)−f(X)]2 (1)
It follows that
MSE ≥Eπ(x)[Var(Y|X)] withequalityiffE(Y|X)=f(X),thebestpredictorofY givenX
under MSE criteria.
E (Y |X ) is the target of our learning procedure. We want an f (X ) that is a good approximation of E (Y |X ) or even f (X ) = E (Y |X ) exactly if possible.
Sarat C. Dass Department of Mathematical and ComInptruotdeurcStcioienntcoesMHaecrhioint-eWLaetatrnUingiversity Malaysia Campus
13/102

The challenges involved
The joint pdf π(x,y) will usually be unknown and therefore E(Y|X) is also unknown.
The first error term on the RHS of (1) is the variance of ε which is inherent noise and hence irreducible.
The second error term on the RHS of (1) is a measure of how well f (X ) approximates E (Y |X ).
This error can be reduced by estimating f from a suitably selected class of functions which mimics the unknown form of E(Y|X).
This component is called the reducible error component – this is the focus for us.
Sarat C. Dass Department of Mathematical and ComInptruotdeurcStcioienntcoesMHaecrhioint-eWLaetatrnUingiversity Malaysia Campus 14/102

Let’s go back to the problem of estimation
Recall the difficulties:
􏲤 The joint pdf π(x , y ) is unknown and hence E (Y |X ) is unknown.
􏲤 Need to choose a class of functions C which can mimic the form of the
unknown E(Y|X).
􏲤 Need a error criteria to perform the estimation of f within C.
Here are the solutions:
􏲤 Obtain a training set (xi,yi), i = 1,2,··· ,n iid from π(x,y).
􏲤 Choose a class of functions C which can reasonably model E (Y |X ),
i.e., the relationship between x and y in the training data set.
􏲤 In the current context, choose the empirical MSE as the criteria for
estimating f ∈ C.
Sarat C. Dass Department of Mathematical and ComInptruotdeurcStcioienntcoesMHaecrhioint-eWLaetatrnUingiversity Malaysia Campus 15/102

Measuring the quality of fit: The empirical MSE criteria
In the regression setting, the most commonly-used measure is the (empirical) mean squared error (MSE), given by
1 􏱜n
MSE = n
based on a training dataset (xi,yi), i = 1,2,··· ,n of size n which are assumed to be iid from π(x,y).
Note that the above (empirical) MSE is an estimate of the population MSE given in (1) since
pop. MSE ≡ Eπ(x,y)(Y − f (X))2 =∧ Eπˆ(x,y)(Y − f (X))2
1 􏱜n
= n
where πˆ(x,y) is the empirical distribution which puts mass 1/n on
each point (xi,yi), i = 1,2,··· ,n.
Sarat C. Dass Department of Mathematical and ComInptruotdeurcStcioienntcoesMHaecrhioint-eWLaetatrnUingiversity Malaysia Campus 16/102
i=1
(yi −f(xi))2 (2)
(yi −f(xi))2 ≡emp. MSE
i=1

Next, choose a class C
In order to estimate f , we have to choose a class of functions C that can reasonably model the relationship between X and Y that we observe in the training dataset.
In Example 2, we can choose C = C1 to be the class of linear functions: f (X) = β0 + β1X, and β0 and β1 are unknown parameters that have to be estimated in order to obtain fˆ.
The MSE criteria to obtain fˆ becomes
where
1 􏱜n
fˆ(X ) = arg min n (yi − f (xi ))2 = βˆ0 + βˆ1 X
f∈C1 i=1
ˆˆ 1􏱜n (β0,β1)=argminn (yi −β0−β1xi)2
β0 ,β1 i =1
Sarat C. Dass Department of Mathematical and ComInptruotdeurcStcioienntcoesMHaecrhioint-eWLaetatrnUingiversity Malaysia Campus
17/102

Least squares regression
Recall that
ˆ 1􏱜n (β0,β1)=argminn (yi −β0−β1xi)2
β0 ,β1 i =1
is precisely the least squares critria you learnt for simple linear regression previously.
The estimates βˆ0 and βˆ1 are the least squares estimates
βˆ0=y ̄−βˆ1x ̄ and
ˆ
Thus, the predictor of Y at given X is
􏰢ni=1(yi − y ̄)(xi − x ̄) β1 = 􏰢ni=1(xi −x ̄)2
fˆ(x)=βˆ +βˆx 01
In other words, the unknown E(Y|X) is approximated by a linear
function of X.
Sarat C. Dass Department of Mathematical and ComInptruotdeurcStcioienntcoesMHaecrhioint-eWLaetatrnUingiversity Malaysia Campus
18/102

Least squares simple linear regression: Example 2
Let’s fit a least squares regression line to the plot of Y versus X in Example 2.
10 12 14 16 18 20 22 10 12 14 16 18 20 22
Education Education
Ask yourself: Is the fit good? Your answer will determine whether C1 is reasonable class for the unknown E(Y|X).
Sarat C. Dass Department of Mathematical and ComInptruotdeurcStcioienntcoesMHaecrhioint-eWLaetatrnUingiversity Malaysia Campus
19/102
Income
20 30 40 50 60 70 80
Income
20 30 40 50 60 70 80

R codes
Here are the R codes used to generate the previous figures:
#Example 2: Income versus years of education
#Fit least squares regression line to scatter plot library(splines)
#Need this library for residual diagnostics
#Read data
income1 <- read.csv("Income1.csv") #Obtain scatter plot #Check out the trend with(income1, plot(Education, Income, type = "p", col = "brown", xlab = "Education", ylab="Income", cex = 2, lwd=3) ) Sarat C. Dass Department of Mathematical and ComInptruotdeurcStcioienntcoesMHaecrhioint-eWLaetatrnUingiversity Malaysia Campus 20/102 R codes (cont.) #Fit least squares regression line lm_fit <- lm(Income ~ Education, data = income1) #out is a lm object which will be used subsequently #Summary of fit analysis summary(lm_fit) #Draw the regression line abline(lm_fit, col = "blue", lwd=3) Sarat C. Dass Department of Mathematical and ComInptruotdeurcStcioienntcoesMHaecrhioint-eWLaetatrnUingiversity Malaysia Campus 21/102 Output of lm fit ## ## Call: ## lm(formula = Income ~ Education, data = income1) ## ## Residuals: ## Min 1Q Median 3Q Max ## -13.046 -2.293 0.472 3.288 10.110 ## ## Coefficients: ## Estimate Std. Error t value Pr(>|t|)
## (Intercept) -39.4463 4.7248 -8.349 4.4e-09 ***
## Education 5.5995 0.2882 19.431 < 2e-16 *** ## --- ## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ## ## Residual standard error: 5.653 on 28 degrees of freedom ## Multiple R-squared: 0.931,Adjusted R-squared: 0.9285 Sara#tC#.DFas-sDsetpaartmteintsotfMiacth:ema3ti7ca7la.nd6ComIonptnruotdeur1cStcioiaennntcoedsMHaec2rhioi8nt-eWLDaetaFtrnU,ingiverspity-MvaalalysiuaeCa:mpu% sample_frac(0.7) #Validation dataset data.frame
valid <- income1 %>% setdiff(train)
Sarat C. Dass Department of Mathematical and ComInptruotdeurcStcioienntcoesMHaecrhioint-eWLaetatrnUingiversity Malaysia Campus
39/102