TA: Winter 2022
…………… . …. …………….. …
MIE1624: Tutorial 6 Bias-variance Trade-off Winter 2022 1 / 18
Bias-variance Trade-off MIE1624:Tutorial6
Copyright By PowCoder代写 加微信 powcoder
Slide Attribution
The following slides contain materials from various sources. Special thanks to the following authors:
Yuehuan He
…………… . …. …………….. …
MIE1624: Tutorial 6
Bias-variance Trade-off
Winter 2022 2 / 18
Bias and Variance Definition
Suppose θˆ is an estimator of θ. Then, we can show the bias of θˆ as:
and the variance of θˆ as:
Bias(θˆ) = E(θˆ) − θ Var(θˆ) = E[(θˆ− E(θˆ))2]
reference: [1]
…………… . …. …………….. …
MIE1624: Tutorial 6
Bias-variance Trade-off Winter 2022 3 / 18
Problems associated with Bias and Variance
High bias Underfitting
The model would be too simple to learn useful trends in the data
The performance of the model on both training and test data would be low
High variance: Overfitting
The model would be too complex to generalize well to unseen data The performance of the model on training data would be much higher than that on test data
…………… . …. …………….. …
MIE1624: Tutorial 6 Bias-variance Trade-off Winter 2022 4 / 18
Dartboard Analogy
…………… . …. …………….. …
MIE1624: Tutorial 6 Bias-variance Trade-off Winter 2022 5 / 18
…………… . …. …………….. …
MIE1624: Tutorial 6 Bias-variance Trade-off Winter 2022 6 / 18
Bias-variance Decomposition
Consider the mean squared error (MSE) of ˆf(x):
…………… . …. …………….. …
MIE1624: Tutorial 6 Bias-variance Trade-off Winter 2022 7 / 18
Consider a target function f(x) = sin(nx) and a data set of size N = 2. We sample x uniformly in [−1, 1) to generate a data set (x1, y1), (x2, y2); and fit the data using one of two models:
H0 : Setofalllinesoftheformh(x)=b;
H1 : Set of all lines of the form h(x) = ax+b.
…………… . …. …………….. …
MIE1624: Tutorial 6 Bias-variance Trade-off Winter 2022 8 / 18
…………… . …. …………….. …
MIE1624: Tutorial 6 Bias-variance Trade-off Winter 2022 9 / 18
Model Complexity
reference: [2]
…………… . …. …………….. …
MIE1624: Tutorial 6 Bias-variance Trade-off Winter 2022 10 / 18
Bias-variance Decomposition in Regression
For any estimator βˆ in regression model:
E[(βˆ − β∗)2] = E[(βˆ − E(βˆ) + E(βˆ) − β∗)2]
= E[(βˆ − E(βˆ))2] + (E(βˆ) − β∗)2 E[(βˆ − E(βˆ))2] is the variance of βˆ;
(E(βˆ) − β∗)2 is the squared bias.
Recall the closed form solution (estimator) for linear regression (βOLS) and
Ridge regression (βλ):
βOLS = (X⊤X)−1(X⊤y), βλ = (X⊤X + λI)−1(X⊤y)
…………… . …. …………….. …
MIE1624: Tutorial 6 Bias-variance Trade-off Winter 2022 11 / 18
Bias-variance Decomposition in Regression
Now consider the variance of Ridge Regression:
…………… . …. …………….. …
MIE1624: Tutorial 6 Bias-variance Trade-off Winter 2022 12 / 18
Proposition (Variance of Ridge Regression Estimator)
Let βλ be the Ridge regression estimator and βOLS be the linear regression estimator, i.e.
Then ∀λ > 0,
βˆ =argmin p∥y−Xβ∥2+λ∥β∥2 λ β∈R 2
E[(βˆ −E(βˆ))2]≤E[(βˆ −E(βˆ ))2] λ λ OLS OLS
Bias-variance Decomposition in Regression
Ridge and estimator for generated data:
see details in ‘Ridge and Lasso – Visualizing Optimal Solutions.ipynb’ from Lecture 5.
…………… . …. …………….. …
MIE1624: Tutorial 6 Bias-variance Trade-off Winter 2022 13 / 18
Bias and Variance in Polynomial Regression
Polynomial Regression:
One of the regression types
The target is an nth degree function of the features [3]
The degree of the function can have a significant effect on model bias/variance
reference: [4]
…………… . …. …………….. …
MIE1624: Tutorial 6 Bias-variance Trade-off Winter 2022 14 / 18
Bias and Variance in Neural Nets
Reference: Yang, Z., Yu, Y., You, C., Steinhardt, J. and Ma, Y., 2020. Rethinking
bias-variance trade-off for generalization of neural networks. arXiv:2002.11328.
…………….. …
…………… . ….
MIE1624: Tutorial 6 Bias-variance Trade-off Winter 2022 15 / 18
Model Selection
How to select a model with the best performance on our available data? How to determine its complexity? How to tune its hyperparameters? We can select the right complexity model in a data-driven/adaptive way. Different Model Selection Techniques:
Cross-validation
Information Criteria – AIC, BIC, Minimum Description Length etc. Best subset selection
Structural Risk Minimization
Complexity Regularization
…………… . …. …………….. …
MIE1624: Tutorial 6 Bias-variance Trade-off Winter 2022 16 / 18
Model Selection Techniques: k-fold Cross-validation
reference:[5]
…………… . …. …………….. …
MIE1624: Tutorial 6 Bias-variance Trade-off Winter 2022 17 / 18
References
https://commons.wikimedia.org/wiki/File:Variance-bias.svg.
https://commons.wikimedia.org/wiki/File: Bias_and_variance_contributing_to_total_error.svg.
https://medium.com/geekculture/ concept-of-machine-learning-polynomial-regression-errors-noise-bias-and-
https://commons.wikimedia.org/wiki/File:Poly-reg-3.png. https://scikit-learn.org/stable/modules/cross_validation.html.
…………… . …. …………….. …
MIE1624: Tutorial 6 Bias-variance Trade-off Winter 2022 18 / 18
程序代写 CS代考 加微信: powcoder QQ: 1823890830 Email: powcoder@163.com