MAST90083 Computational Statistics & Data MiningNonparametric Regression
Tutorial & Practical 7: Nonparametric Regression
Question 1
Consider the ordinary nonparametric regression model
yi = f(xi) + �i; 1 ≤ i ≤ n
where yi ∈ R, xi ∈ R, �i ∈ R ∼ N (0, σ2) and are i.i.d. For approximating f we propose to
use the linear spline model.
1. Provide the form of this linear spline model and define the set of linear parameters
2. Derive its matrix form and the associated penalized spline fitting criterion
3. To what this criterion corresponds when considering the coefficients of the truncated
linear functions as random with cov(u) = σ2uI where u is the vector of coefficients of
the truncated linear function.
4. Derive the expression for the penalized least squares estimator of the unknown param-
eters of the model
5. Provide the associated expression for the best fitted values
6. Provide a model form which the estimation of the coefficients of the polynomial func-
tions is straightforwardly obtained
7. Derive the estimator of these parameters and their covariance
Question 2
1. Provide the expression of the Nadaraya-Watson kernel estimator and discuss its singu-
larity condition
2. Is the Nadaraya-Watson estimator with a Gaussian Kernel differentiable?
3. Is the Nadaraya-Watson estimator with the Epanechnikov Kernel differentiable?
Question 3
Let’s consider a set of observations generated according to the model
yi = f(xi) + �i; 1 ≤ i ≤ n
where f : [0, 1]→ R, the pairs (xi, yi) , i = 1, …, n are observed and �i are i.i.d. with E(�i) = 0
and E (�2i ) = σ
2
� .
1
MAST90083 Computational Statistics & Data MiningNonparametric Regression
Assuming f ∈ L2 [0, 1] and an orthonormal basis {ρj}
∞
j=1
of L2 [0, 1] then
f(x) =
∞∑
j=1
θjρj(x),
where
θj =
∫ 1
0
f(x)ρj(x)dx.
The projection estimator for f is given by
f̂nN(x) =
N∑
j=1
θ̂jρj(x),
where
θ̂j =
1
n
n∑
i=1
yiρj(xi).
1. Which parameter plays the role of the smoothing parameter in the projection estimator
2. Show that f̂nN is a linear estimator in y
′
is
3. Show that E
(
θ̂j
)
= θj + rj
4. Show that E
[(
θ̂j − θj
)2]
=
σ2�
n
+ r2j
5. Provide the expression of the MISE = E
[
‖f̂nN − f‖22
]
6. Discuss the effect of N on the MISE
2