Resampling methods
Faculty of Information Technology, Monash University, Australia
FIT5149 week 5
(Monash) FIT5149 1 / 35
Outline
1 Motivation
2 Cross-Validation
The Validation Set Approach Leave-One-Out Cross
k-fold Cross Validation
The Bootstrap
3 Summary
(Monash)
FIT5149 2 / 35
Motivation
Bias-Variance
(Monash) FIT5149 3 / 35
Motivation
Motivation
Motivation
Motivation & Introduction
To draw many samples from the training set and refit the model on each sample to get better information on the model
Extra information that is not available from fitting the model only once To examine how the resulting fits are different
Two the most commonly used resampling methods:
… Cross Validation
Be used to estimate the test error associated with a given statistical learning
method
in order to evaluate its performance, or to select the appropriate level of
flexibility
… Bootstrap
To provide a measure of accuracy of a parameter estimate or of a given
statistical learning method
The process of evaluating the performance of a model is known as model assessment
The process of selecting the proper level of flexibility for a model is known as model selection
Lan-Asef (Monash) FIT5149 5 / 40 (Monash) FIT5149 4 / 35
Motivation
Test and Training Errors
Test and Training Errors
Cross-Validation
Test error rate
… Is the average error that results from using a statistical learning method to predict the response on a new observation
… That is, a measurement that was not used in training the method.
… Given a data set, the use of a particular statistical learning method is
warranted if it results in a low test error.
… The test error can be easily calculated if a designated test set is available.
… Unfortunately, this is usually not the case
Training error rate
… Can be easily calculated by applying the statistical learning method to the observations used in its training.
… The training error rate often is quite different from the test error rate,
… And in particular the former can dramatically underestimate the latter.
Lan-Asef (Monash) FIT5149 (Monash) FIT5149
6 / 40
5 / 35
Motivation
Test and Training Errors
Cross-Validation
Test and Training Errors
Lan-Asef (Monash) FIT5149 7 / 40 (Monash) FIT5149 5 / 35
Motivation
Test and Training Errors
Cross-Validation
Test and Training Errors
What if there is not a large enough test set to estimate the test error rate!?
… A number of techniques can be used to estimate this quantity using the available training data.
… In this section, we consider a class of methods that estimate the test error rate by holding out a subset of the training observations from the fitting process,
… And then applying the statistical learning method to those held out observations.
Distinguish between
… Quantitative response variable: regression models
… Qualitative response variable: classification models
Lan-Asef (Monash) FIT5149 (Monash) FIT5149
8 / 40
5 / 35
Cross-Validation
The Validation Set Approach
The Validation Set Approach
Cross-Validation The Validation Set Approach
Aim: to estimate the test error associated with fitting a particular statistical learning method on a set of observations
The validation set approach
… randomly dividing the available set of observations into two parts: a training set and a validation set or hold-out set
… The model is fit on the training set, and the fitted model is used to predict the responses for the observations in the validation set.
… The resulting validation set error rate (typically assessed using MSE in the case of a quantitative response) provides an estimate of the test error rate.
!”#”$””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””%”
&””##””!$”””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””‘!”
Lan-Asef (Monash) FIT5149 9 / 40 (Monash) FIT5149 6 / 35
The Validation Set Approach
Cross-Validation
The Validation Set Approach
Error from Validation Set
Cross-Validation The Validation Set Approach
L: Error estimates for a single 50-50 split into training and validation
R: validation method was repeated ten times 50-50 random splits
Lan-Asef (Monash) FIT5149
(Monash) FIT5149 7 / 35
10 / 40
Error from Validation Set
Cross-Validation
The Validation Set Approach
Error from Validation Set
Cross-Validation The Validation Set Approach
Left: Error estimates for a single split into training and validation data sets Right: validation method was repeated ten times, each time using a different random split
Based on the variability among these curves, all that we can conclude with any confidence is that the linear fit is not adequate for this data.
Lan-Asef (Monash) FIT5149 11 / 40 (Monash) FIT5149 7 / 35
Error from Validation Set
Cross-Validation
The Validation Set Approach
Disadvantages
Disadvantages
Cross-Validation The Validation Set Approach
The validation set approach is conceptually simple and is easy to implement. But it has two potential drawbacks
1 the validation estimate of the test error rate can be highly variable, depending on precisely which observations are included in the training set and which observations are included in the validation set
2 In the validation approach, only a subset of the observations (those that are included in the training set rather than in the validation set) used to fit the model
Since statistical methods tend to perform worse when trained on fewer observations,
This suggests that the validation set error rate may tend to overestimate the test error rate for the model fit on the entire data set.
Estimators
… Point estimator
… Interval estimator
… Unbiased estimator
Lan-Asef (Monash)
FIT5149
12 / 40
(Monash)
FIT5149 8 / 35
Cross-Validation
Leave-One-Out Cross
Leave-One-Out Cross-Validation
Leave-One-Out Cross-Validation (LOOCV)
Cross-Validation Leave-One-Out Cross-Validation (LOOCV)
Involves splitting the set of observations into two parts
A single observation x,y is used for the validation set, and the
remaining observations x2,y2 ,…, xn,yn make up the training set The model is fit on the n training observations, and a prediction yö is
made for the excluded observation, using its value x
x ,y was not used in the fitting process, MSE = y yö 2 provides an
approximately unbiased estimate for the test error.
But even though MSE is unbiased for the test error, it is a poor estimate
because it is highly variable
It is based upon a single observation x,y
We can repeat the procedure by selecting x2,y2 for the validation data,
and for training x,y , x3,y3 ,…, xn,yn MSE = y yö 2
Lan-Asef (Monash) FIT5149 (Monash) FIT5149
and computing
222
13 / 40
9 / 35
Cross-Validation
Leave-One-Out Cross
Leave-One-Out Cross-Validation
Leave-One-Out Cross-Validation (LOOCV)
Cross-Validation Leave-One-Out Cross-Validation (LOOCV)
Repeating this approach n times produces n squared errors,
MSE, . . . , MSEn. The LOOCV estimate for the test MSE is the average of these n test error estimates:
Xn CVn=n MSEi
i=
Lan-Asef (Monash) FIT5149
(Monash) FIT5149 9 / 35
14 / 40
Cross-Validation
Leave-One-Out Cross
Cross-Validation Leave-One-Out Cross-Validation (LOOCV) Major Advantages of LOOCV
LOOCV: Advantages
It has far less bias
… We repeatedly fit the statistical learning method using training sets that contain n observations
… Almost as many as are in the entire data set
… In the validation set approach, in which the training set is typically around
half the size of the original data set
… The LOOCV approach tends not to overestimate the test error rate as much
as the validation set approach does
Performing LOOCV multiple times will always yield the same results
… The validation approach will yield different results when applied repeatedly due to randomness in the training/validation set splits
… There is no randomness in the training/validation set splits.
Lan-Asef (Monash) FIT5149 15 / 40 (Monash) FIT5149 10 / 35
Cross-Validation
Leave-One-Out Cross
Test LOOCV on the Auto Data
Test of LOOCV on the Auto data
Cross-Validation Leave-One-Out Cross-Validation (LOOCV)
To obtain an estimate of the test set MSE
From fitting a linear regression model to predict mpg using polynomial functions of horsepower
The LOOCV error curve
Lan-Asef (Monash) FIT5149 (Monash) FIT5149
16 / 40
11 / 35
Cross-Validation
Leave-One-Out Cross
LOOCV: Disadvantages
Cross-Validation Leave-One-Out Cross-Validation (LOOCV)
LOOCV – Disadvantage
LOOCV has the potential to be expensive to implement the model has to be fit n times
If n is large and each individual model is slow to fit!! However,
… LOOCV is a very general method, and can be used with any kind of predictive modeling
… we could use it with logistic regression or linear discriminant analysis
Lan-Asef (Monash) FIT5149 (Monash) FIT5149
17 / 40
12 / 35
Cross-Validation
k-fold Cross Validation
k-fold CV
Cross-Validation k-Fold Cross-Validation
An alternative to LOOCV is k-fold CV
Randomly dividing the set of observations into k groups, or folds, of approximately equal size
The first fold is treated as a validation set, and the method is fit on the remaining k folds.
The mean squared error, MSE, is then computed on the observations in the held-out fold
This procedure is repeated k times
Each time, a different group of observations is treated as a validation set The k-fold CV estimate is computed by averaging these values,
Xk CVk=k MSEi
i=
Lan-Asef (Monash) FIT5149 (Monash) FIT5149
18 / 40
13 / 35
k-fold CV
Cross-Validation
k-fold Cross Validation
5-fold CV
Cross-Validation k-Fold Cross-Validation
A set of n observations is randomly split into five non-overlapping groups Each of these fifths acts as a validation set (shown in beige),
And the remainder as a training set (shown in blue)
The test error is estimated by averaging the five resulting MSE estimates
!”#”$””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””%”
!!”&'”(“””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””)&”
!!”&'”(“””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””)&”
!!”&'”(“””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””)&”
!!”&'”(“””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””)&”
!!”&'”(“””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””””)&”
Lan-Asef (Monash) FIT5149 19 / 40 (Monash) FIT5149 14 / 35
5-fold CV
Cross-Validation
k-fold Cross Validation
k-fold CV
Cross-Validation k-Fold Cross-Validation
LOOCV is a special case of k-fold CV in which k is set to equal n
In practice, one typically performs k-fold CV using k = 5 or k = à
LOOCV requires fitting the statistical learning method n times
Performing 10-fold CV requires fitting the learning procedure only ten times
Other non-computational advantages to performing k-fold CV, involve bias-variance trade-off
Lan-Asef (Monash) FIT5149 20 / 40 (Monash) FIT5149 15 / 35
k-fold CV
Cross-Validation
k-fold Cross Validation
Error from k-fold CV
Error from k-fold
Cross-Validation k-Fold Cross-Validation
Nine different 10-fold CV estimates for the Auto data set
Each resulting from a different random split of the observations into 10 folds There is some variability in the CV estimates as a result of the variability in how the observations are divided into ten folds
Variability is lower than the validation set approach in estimating test error
Lan-Asef (Monash) FIT5149 21 / 40 (Monash) FIT5149 16 / 35
Cross-Validation
k-fold Cross Validation
k-fold CV in Application
Cross-Validation k-Fold Cross-Validation
When we examine real data, we do not know the true test MSE
So it is difficult to determine the accuracy of the cross-validation estimate
If we examine simulated data, then we can compute the true test MSE, and can thereby evaluate the accuracy of our cross-validation results.
Lan-Asef (Monash) FIT5149 22 / 40 (Monash) FIT5149 17 / 35
In Application
Cross-Validation
k-fold Cross Validation
k-fold CV: Example
Cross-Validation k-Fold Cross-Validation
True and estimated test MSE for the simulated data sets Figures 2.9 (left), 2.10 (center), and 2.11 (right)
The true test MSE is shown in blue
The LOOCV estimate is shown as a black dashed line, The 10-fold CV estimate is shown in orange
The crosses indicate the minimum of each of the MSE curves.
Lan-Asef (Monash)
FIT5149
23 / 40
2 5 10 20
Flexibility
2
5 10 20
Flexibility
2 5 10 20
Flexibility
(Monash)
FIT5149
18 / 35
Example
Mean Squared Error
0.0 0.5 1.0 1.5 2.0 2.5 3.0
Mean Squared Error
0.0 0.5 1.0 1.5 2.0 2.5 3.0
Mean Squared Error
0 5 10 15 20
Cross-Validation
k-fold Cross Validation
Cross-Validation k-Fold Cross-Validation Model Assessment and Model Selection
When we perform cross-validation, our goal might be to determine how well a given statistical learning procedure can be expected to perform on independent data
the actual estimate of the test MSE is of interest
But at other times we are interested only in the location of the minimum point in the estimated test MSE curve
This is because we might be performing cross-validation on a number of statistical learning methods, or on a single method using different levels of flexibility,
In order to identify the method that results in the lowest test error
The location of the minimum point in the estimated test MSE curve is important,
But the actual value of the estimated test MSE is not
Despite the fact that they sometimes underestimate the true test MSE, all of the CV curves come close to identifying the correct level of flexibility
Lan-Asef (Monash) FIT5149 24 / 40 (Monash) FIT5149 19 / 35
Model Assessment and Model Selection
Cross-Validation
k-fold Cross Validation
Bias-Variance
Cross-Validation Bias-Variance Trade-Off for k-Fold Cross-Validation
k-fold CV with k < n has a computational advantage to LOOCV
k-fold gives more accurate estimates of the test error rate than does LOOCV
Validation set approach can lead to overestimates of the test error rate Since the training set used to fit the model contains only half the observations of the entire data set
LOOCV will give approximately unbiased estimates of the test error
Each training set contains n observations, which is almost as many as the number of observations in the full data set
Performing k-fold CV for, say, k = 5 or k = à will lead to an intermediate level of bias
Since each training set contains k n/k observations
Fewer than in the LOOCV approach, but substantially more than in the validation set approach
From the perspective of bias reduction, it is clear that LOOCV is to be preferred to k-fold CV
Lan-Asef (Monash) FIT5149 (Monash) FIT5149
25 / 40
20 / 35
Bias-Variance
Cross-Validation
k-fold Cross Validation
Bias-Variance
Cross-Validation Bias-Variance Trade-Off for k-Fold Cross-Validation
Bias-Variance
Bias is not the only source for concern in an estimating
We must also consider the procedure’s variance
LOOCV has higher variance than does k-fold CV with k < n
LOOCV, averaging the outputs of n fitted models, each of which is trained on an almost identical set of observations;
Therefore, these outputs are highly (positively) correlated with each other
k-fold CV with k < n, we are averaging the outputs of k fitted models that are somewhat less correlated with each other
The test error estimate resulting from LOOCV tends to have higher variance than does the test error estimate resulting from k-fold CV
There is a bias-variance trade-off associated with the choice of k in k-fold cross-validation
Using k = 5 or k = à, as these values have been shown empirically to yield test error rate estimates that suffer neither from excessively high bias nor from very high variance.
Lan-Asef (Monash) FIT5149 26 / 40 (Monash) FIT5149 21 / 35
Cross-Validation
k-fold Cross Validation
Classification
Cross-Validation Cross-Validation on Classification Problems
Cross-validation can also be a very useful approach in the classification setting when Y is qualitative
Rather than using MSE to quantify test error, we instead use the number of misclassified observations
The LOOCV error rate Xn CVn= Erri
n i=
The k-fold CV error rate and validation set error rates are defined
Err = I y 6= yö iii
analogously.
Lan-Asef (Monash) FIT5149 (Monash) FIT5149
27 / 40
22 / 35
Classification
Cross-Validation
k-fold Cross Validation
Example
Cross-Validation Cross-Validation on Classification Problems
Top-left: the black solid line shows the estimated decision boundary from fitting a standard logistic regression model
This is simulated data, we can compute the true test error rate
Which is 0.201 and so is substantially larger than the Bayes error rate of 0.133
Logistic regression does not have enough flexibility to model the Bayes decision boundary
In logistic regression, we get non-linear decision boundary by using polynomial functions of the predictors
We can fit a quadratic logistic regression model
logÅ p ã= à+ X+ 2X2+ 3X2+ 4X2 p
Top-right displays the resulting decision boundary, which is now curved The test error rate has improved only slightly, to 0.197
Lan-Asef (Monash) FIT5149 (Monash) FIT5149
28 / 40
23 / 35
Example
Example
Cross-Validation
Cross-Validation
k-fold Cross Validation
Cross-Validation on Classification Problems
Example
Lan-Asef (Monash)
FIT5149
29 / 40
(Monash)
FIT5149
24 / 35
o
o
ooooo oo ooo o oo
o o o o oo
o o o
o
o
ooooo oo ooo o oo
o o o o oo
o o o
Degree=1
Degree=2
oo oo oo o o oo o o
oo oo o o oo o o o oo o
oo
o ooo o o o ooo o
o
oo oo oooo o oo oo oooo o
o oo o o o o o oo o o o o oooooooo o oooooooo o
o oooo o o oooo o
o oo o o o oo o o
oo ooo oo o oo ooo oo o
oo
o o o o o
ooo o
o oo o o o o o o
oo ooo o
o o o o o o oo o o o o o
oooo
o
o
ooooo ooooo
o o
oo o o
o oo o ooo o
oo o o o oo o ooo o
o
o
oo ooo
oo ooo
o ooooo oo ooo o oo
o
o o o oo o
o o
o oo o
oo o o ooo
o oo o oo
oo
o ooo
o o oooo o ooo
o o ooo oo o oo o
o o oooooooo
oo
oo oo oo
Degree=3 Degree=4
oo oo oo o o oo o o
oo oo o o oo o o o oo o
o
o o o oo o
oo
o ooo o o o ooo o
o
oo oo oooo o oo oo oooo o
o oo o o o o o oo o o o o oooooooo o oooooooo o
o oooo o o oooo o
o oo o o o oo o o
oo ooo oo o oo ooo oo o
oo
o o o o o
ooo o
o oo o o o o o o
oo ooo o
o o o o o o oo o o o o o
oooo
o
o
ooooo ooooo
o o
oo o o
o oo o ooo o
oo o o o oo o ooo o
o
o
oo ooo
oo ooo
o ooooo oo ooo o oo
o o
o oo o
oo o o ooo
o oo o oo
oo
o ooo
o o oooo o ooo
o o ooo oo o oo o
o o oooooooo
oo
oo oo oo
oo o oo o
o o o oooo o
oo o oo o
o o o oooo o
Cross-Validation
k-fold Cross Validation
Example
Cross-Validation Cross-Validation on Classification Problems
Example
A larger improvement is in the bottom-left, we have fit a logistic regression model involving cubic polynomials of the predictors
Test error rate has decreased to 0.160
Going to a quartic polynomial (bottom-right) slightly increases the test error
In practice, for real data, the Bayes decision boundary and the test error rates are unknown
To decide on decide between the four logistic regression models We can use cross-validation in order to make this decision
Lan-Asef (Monash) FIT5149 (Monash) FIT5149
30 / 40
25 / 35
Cross-Validation
k-fold Cross Validation
Example
Cross-Validation Cross-Validation on Classification Problems
Classification Example
Test error (brown), training error (blue), and 10-fold CV error (black) on the two-dimensional classification data
Left: Logistic regression using polynomial functions of the predictors The order of the polynomials used is displayed on the x-axis.
Right: The KNN classifier with different values of K, the number of neighbors used in the KNN classifier
Lan-Asef (Monash)
FIT5149
31 / 40
2
4 6 8
Order of Polynomials Used
10
0.01 0.02
0.05 0.10 0.20
1/K
0.50 1.00
(Monash)
FIT5149
26 / 35
Error Rate
0.12 0.14 0.16 0.18 0.20
Error Rate
0.12 0.14 0.16 0.18 0.20
Cross-Validation
The Bootstrap
Outline
1 Motivation
2 Cross-Validation
The Validation Set Approach Leave-One-Out Cross
k-fold Cross Validation
The Bootstrap
3 Summary
(Monash)
FIT5149 27 / 35
Cross-Validation
The Bootstrap
The Bootstrap
The Bootstrap
The Bootstrap
Used to quantify the uncertainty associated with a given estimator or statistical learning method
Example: can be used to estimate the standard errors of the coefficients from a linear regression fit
For linear regression, not a big deal! why?
It can be easily applied to a wide range of statistical learning methods
Lan-Asef (Monash) FIT5149 (Monash) FIT5149
32 / 40
28 / 35
Cross-Validation
The Bootstrap
A Toy Example
The Bootstrap
A toy Example
We wish to determine the best investment allocation
Invest a fixed sum of money in two financial assets that yield returns of X
and Y (random quantities)
Invest a fraction of ↵ in X and ↵ in Y
There is variability associated with the returns on these two assets Choose ↵ to minimize the total risk, or variance, of our investment
It is proven that
minimizeVar ↵X+ ↵Y ↵ = Y2 X Y
X2 + Y2 2 X Y
Where x2=VarX, x2=VarX, XY =CovX,Y
Wedon’tknow x2 =Var X , x2 =Var X , XY =Cov X,Y inreality! What we can do then??
Lan-Asef (Monash) FIT5149 33 / 40 (Monash) FIT5149 29 / 35
Cross-Validation
The Bootstrap
A Toy Example
The Bootstrap
Weestimatethem: öX2 =Var X , öX2 =Var X , öXY =Cov X,Y Using a data set that contains past measurements for X and Y
We can then estimate the value of ↵ that minimizes the variance of our
investment
↵ö = ö Y2 ö X Y
ö X2 + ö Y2 2 ö X Y
Simulated 100 pairs of returns for the investments X and Y These return are used to estimate
x2=VarX, x2=VarX, XY =CovX,Y These estimates are substituted to find ↵ö
Lan-Asef (Monash) FIT5149 (Monash) FIT5149
34 / 40
29 / 35
A toy Example
Cross-Validation
The Bootstrap
A Toy Example
The Bootstrap
A toy Example
Each panel displays 100 simulated returns for investments X and Y From left to right and top to bottom, the resulting estimates for ↵ are 0.576, 0.532, 0.657, and 0.651
−2 −1 0 1 2 −2 −1 0 1 2
XX
Lan-Asef (Monash)
FIT5149
35 / 40
−3 −2 −1 0 1 2 −2 −1 0 1 2 3
XX
(Monash)
FIT5149
29 / 35
YY
−3 −2 −1 0 1 2 −2 −1 0 1 2
YY
−3 −2 −1 0 1 2 −2 −1 0 1 2
Cross-Validation
The Bootstrap
Accuracy of α ↵
The Bootstrap
Natural question: quantify the accuracy of our estimate of ↵ To estimate the standard deviation of ↵ö:
... the process of simulating 100 paired observations of X and Y
... and estimating ↵ using
... 1,000 times
... we obtained ↵ö, à à à , ↵öààà
Lan-Asef (Monash)
FIT5149
36 / 40
0.4 0.5
0.6
0.7 0.8 0.9 0.3 0.4 0.5 0.6 0.7 0.8 0.9 True
↵↵
Bootstrap
(Monash)
FIT5149
30 / 35
Accuracy of
0 50 100 150 200
0 50 100 150 200
↵
0.3 0.4 0.5 0.6 0.7 0.8 0.9
Cross-Validation
The Bootstrap
Accuracy of α
The Bootstrap
For these simulations the parameters were set x2 = , x2 = , XY = .25
we know that ↵ = à.6 (solid vertical line on the histogram)
The mean is X ààà
↵¬ = ààà ↵ör = à.5996 r=
the standard deviation of the estimates
ààà vu X
tààà ↵ör ↵¬ 2 =à.à83 r=
This gives us a very good idea of the accuracy of ↵ö : SE ↵ö So roughly speaking
⇡ à.à83
... we would expect ↵ö to differ from ↵ by approximately 0.08, on average.
... for a random sample from the population
Lan-Asef (Monash) FIT5149 (Monash) FIT5149
37 / 40
31 / 35
Accuracy of ↵
Cross-Validation
The Bootstrap
In Practice
The Bootstrap
In practice the procedure for estimating SE ↵ö outlined above cannot be applied
because for real data we cannot generate new samples from the original population
the bootstrap approach allows us to use a computer to emulate the process of obtaining new sample sets
so that we can estimate the variability of ↵ö without generating additional samples
Rather than repeatedly obtaining independent data sets from the population,
we instead obtain distinct data sets by repeatedly sampling observations from the original data set
Lan-Asef (Monash) FIT5149 38 / 40 (Monash) FIT5149 32 / 35
In Practice
Cross-Validation
The Bootstrap
Bootstrap Example
The Bootstrap
a small sample containing n = 3 observations
Each bootstrap data set contains n observations, sampled with replacement from the original data set
Each bootstrap data set is used to obtain an estimate of ↵
Obs
1
2
3
X
4.3
2.1
5.3
Y
2.4
Obs
3
1
3
Obs
2
X
5.3
4.3
5.3
X
2.1
Y
2.8
2.4
Z*1 Z*2
!!
!!
!! !!
αˆ * 1
αˆ * 2
!!
!!
2.8
Y
1.1
1.1
2.8
3
1
5.3
4.3
!! !! !!
2.8
2.4
Obs
X
Y
2
2.1
1.1
2
2.1
1.1
1
4.3
2.4
Lan-Asef (Monash)
FIT5149
39 / 40
(Monash)
FIT5149
33 / 35
Original Data (Z)
!Z*B !!
!!
!!
αˆ * B
Bootstrap Example
Cross-Validation
The Bootstrap
Bootstrap Example
The Bootstrap
a simple data set, which we call Z, that contains only n = 3 observations We randomly select n observations from the data set in order to produce a
bootstrap data set
The sampling is performed with replacement
the same observation can occur more than once in the bootstrap data set
Repeat B times
... different bootstrap data sets Z⇤, Z⇤2, à à à , Z⇤B
... estimates of ↵ are ↵ö⇤, ↵ö⇤2, à à à , ↵ö⇤B
... standard error of these bootstrap estimates
vuXBÇ XB å2 S E B ↵ö = t B ↵ö ⇤ r B ↵ö ⇤ r 0
r= r0=
... This serves as an estimate of the standard error of ↵ö estimated from the
original data set
Lan-Asef (Monash) FIT5149 (Monash) FIT5149
40 / 40
34 / 35
Bootstrap Example
Summary
Summary
Cross Validation Bootstrap Reading materials:
"Resampling Methods", Chapter 5 of "Introduction to Statistical Learning", 6th edition
References:
Figures in this presentation were taken from "An Introduction to Statistical Learning, with applications in R" (Springer, 2013) with permission from the authors: G. James, D. Witten, T. Hastie and R. Tibshirani
Some of the slides are reproduced based on the slides from T. Hastie and R. Tibshirani
(Monash) FIT5149 35 / 35