MAST90083 Computational Statistics & Data Mining Linear Regression
Tutorial & Practical 3: Ridge Regression
Question 1
Given the model
y = Xβ + �
where y ∈ Rn, X ∈ Rn×p is of rank r ≤ p < n and � ∈ Rn ∼ N (0, σ2In). Let β̂ be the
estimate of β obtained by least square estimation and let X = SΣQ>, where S is an n × n
matrix, Σ = diag (σ1, …, σr) is a diagonal matrix and Q is p× p matrix, be the singular value
decomposition of X
1. Using the singular value decomposition of X, write the least square estimator β̂ of β
as a sum of individual components bi along qi, the columns vectors of Q
2. Derive the expression of the ridge regression estimator and write its expression using bi
3. Provide the form of the shrinkage functions of the singular values σi that characterize
the ridge repression and principal component regression estimators
4. Derive the bias of the ridge regression estimator
5. Derive the variance covariance of the ridge regression estimator and compare the ob-
tained variance with the variance covariance of the least square estimator
6. Provide the distribution of the ridge regression estimator
7. Derive the expression of the mean square error of the ridge regression estimator
8. Derive the expressions of the ridge regression estimator, its bias, variance, mean square
error in the case of orthonormal design matrix X
9. Obtain the value of the regularization parameter that minimizes the mean square error
in the case of orthonormal design matrix X
10. Derive the expression of the square norm of the ridge regression estimator and discuss
its behavior when the regularization parameter is decreasing
Question 2
Let
y = Xβ + �
where y ∈ Rn, X ∈ Rn×p is of rank r ≤ p and � ∈ Rn ∼ N (0, σ2In).
1. Derive the degrees of freedom of the least square estimator
2. Provide its expression in the case of orthonormal design matrix X
3. Derive the degrees of freedom of the ridge regression estimator
4. Provide its expression in the case of orthonormal design matrix X
1