Problem 5.1
CS5487 Problem Set 5
Non-parametric estimation and clustering
Antoni Chan Department of Computer Science City University of Hong Kong
Kernel density estimators
Bias and variance of the kernel density estimator
In this problem, we will derive the bias and variance of the kernel density estimator. Let X = {x1, · · · , xn} be the r.v. samples, drawn independently according to the true density p(x).
(a) Show that the mean of the estimator is
EX[pˆ(x)]=Z p(μ)k ̃(x μ)dμ=p(x)⇤k ̃(x), (5.1)
where ⇤ is the convolution operator. What does this tell you about how the KDE is biased? (b) Show that the variance of the estimator is bounded by
varX (pˆ(x)) 1 max(k(x))E[pˆ(x)]. nhd x
Hint: the following properties will be helpful:
var(x) = E[x2] (E[x])2 E[x2],
k ✓ x xi ◆ max k(x), hx
and Problem 1.4.
………
Problem 5.2 Mean and variance of a kernel density estimate
(5.2)
(5.3) (5.4)
In this problem, we will study the mean and variance of the kernel density estimate, i.e., the distribution pˆ(x). Let X = {x1, · · · , xn} be the set of samples, and k ̃(x) be the kernel with bandwidth included. The estimated probability distribution is
1 Xn
pˆ(x) = n k ̃(x xi).
i=1
Suppose that the kernel function k ̃(x) has zero mean and covariance H, i.e.,
(5.5)
(5.6) (5.7)
E ̃[x] = Z k ̃(x)xdx = 0, kZ ̃T
covk ̃(x)= k(x)(x Ek ̃[x])(x Ek ̃[x]) dx=H. 38
(a) Show that the mean of the distribution pˆ(x) is the sample mean of X, Z 1Xn
μˆ = Epˆ[x] = pˆ(x)xdx = n xi. i=1
(5.8)
(5.9)
(b) Show that the covariance of the distribution pˆ(x) is ˆ 1Xn
⌃ = covpˆ(x) = H + n (xi μˆ)(xi μˆ)T , i=1
where the second term on the right hand side is the sample covariance.
(c) What does this tell you about the properties of the kernel density estimate pˆ(x)? How does this relate to the bias of the kernel density estimator?
………
Problem 5.3 KDE with Gaussians
Consider the kernel function k(x) = N(x|0,1), and samples X = {x1,…,xn} generated from a Gaussian, p(x) = N (x|μ, 2). Show that the kernel density estimate,
1Xn ✓x xi◆ pˆ(x) = nhd k h ,
(5.10)
i=1
(c) bias(pˆ(x)) = p(x) E [pˆ(x)] ⇡ h2 h1 (x μ)2 i p(x). X 2 2 2
has the following properties, for small h:
(a) EX[pˆ(x)]=N(x|μ, 2+h2).
(b) varX(pˆ(x)]⇡ 1p p(x). 2nh ⇡
(d) Setting h as a function of n, h = a/pn, what is the convergence rate of the bias and variance of the estimator, in terms of the number of samples n? How does the convergence rate compare with that of the ML estimator for a Gaussian?
……… Problem 5.4 KDE with exponential kernel
Let the true density p(x) ⇠ U (0, a) be a uni(form density from 0 to a. Let the kernel function be k(x) = e x, x > 0 (5.11)
0, otherwise. (a) Show that the mean of the kernel density estimator is
E[pˆ(x)] =
8><>: 0 , x < 0
1 (1 e x/h), 0 x a (5.12)
1(ea/h 1)e x/h, ax. a
39
a
(b) PlotE[pˆ(x)]versusxfora=1andh={1,1, 1 }. 4 16
(c) Howsmalldoeshneedtobetohavelessthan1%biasover99%oftherange0