Problem 5.1
CS5487 Problem Set 5
Non-parametric estimation and clustering
Department of Computer Science City University of
Copyright By PowCoder代写 加微信 powcoder
Kernel density estimators
Bias and variance of the kernel density estimator
In this problem, we will derive the bias and variance of the kernel density estimator. Let X = {x1, · · · , xn} be the r.v. samples, drawn independently according to the true density p(x).
(a) Show that the mean of the estimator is
EX[pˆ(x)]=Z p(μ)k ̃(x μ)dμ=p(x)⇤k ̃(x), (5.1)
where ⇤ is the convolution operator. What does this tell you about how the KDE is biased? (b) Show that the variance of the estimator is bounded by
varX (pˆ(x)) 1 max(k(x))E[pˆ(x)]. nhd x
Hint: the following properties will be helpful:
var(x) = E[x2] (E[x])2 E[x2],
k ✓ x xi ◆ max k(x), hx
and Problem 1.4.
Problem 5.2 Mean and variance of a kernel density estimate
(5.3) (5.4)
In this problem, we will study the mean and variance of the kernel density estimate, i.e., the distribution pˆ(x). Let X = {x1, · · · , xn} be the set of samples, and k ̃(x) be the kernel with bandwidth included. The estimated probability distribution is
pˆ(x) = n k ̃(x xi).
Suppose that the kernel function k ̃(x) has zero mean and covariance H, i.e.,
(5.6) (5.7)
E ̃[x] = Z k ̃(x)xdx = 0, kZ ̃T
covk ̃(x)= k(x)(x Ek ̃[x])(x Ek ̃[x]) dx=H. 38
(a) Show that the mean of the distribution pˆ(x) is the sample mean of X, Z 1Xn
(b) Show that the covariance of the distribution pˆ(x) is ˆ 1Xn
has the following properties, for small h:
(a) EX[pˆ(x)]=N(x|μ, 2+h2).
(b) varX(pˆ(x)]⇡ 1p p(x). 2nh ⇡
μˆ = Epˆ[x] = pˆ(x)xdx = n xi. i=1
⌃ = covpˆ(x) = H + n (xi μˆ)(xi μˆ)T , i=1
where the second term on the right hand side is the sample covariance.
(c) What does this tell you about the properties of the kernel density estimate pˆ(x)? How does this relate to the bias of the kernel density estimator?
Problem 5.3 KDE with Gaussians
Consider the kernel function k(x) = N(x|0,1), and samples X = {x1,…,xn} generated from a Gaussian, p(x) = N (x|μ, 2). Show that the kernel density estimate,
1Xn ✓x xi◆ pˆ(x) = nhd k h ,
(c) bias(pˆ(x)) = p(x) E [pˆ(x)] ⇡ h2 h1 (x μ)2 i p(x). X 2 2 2
(d) Setting h as a function of n, h = a/pn, what is the convergence rate of the bias and variance of the estimator, in terms of the number of samples n? How does the convergence rate compare with that of the ML estimator for a Gaussian?
……… Problem 5.4 KDE with exponential kernel
Let the true density p(x) ⇠ U (0, a) be a uni(form density from 0 to a. Let the kernel function be k(x) = e x, x > 0 (5.11)
0, otherwise. (a) Show that the mean of the kernel density estimator is
8><>: 0 , x < 0
E[pˆ(x)] = a1 (1 e x/h), 0 x a (5.12)
a1(ea/h 1)e x/h, ax. 39
(b) PlotE[pˆ(x)]versusxfora=1andh={1,1, 1 }. 4 16
(c) Howsmalldoeshneedtobetohavelessthan1%biasover99%oftherange0