Chapter 5
Consistency and Limiting Distributions
5.2 Convergence in Distribution
1/44
Boxiang Wang
Chapter 5 STAT 4101 Spring 2021
Outline
1 Def. of convergence in distribution, from slide 3 to slide 12. Question: Recallthecentrallimittheorem,
X ̄ − μ
Zn = σ/√n is approximately N(0, 1). (⋆)
What does ”approximately” mean?
2 Useful results, from slide 13 to slide 17.
Question: Whenσisunknown,whyuseSinstead?Slutskytheorem. Detour: Convergenceindistributionvsconvergenceinprobability.
3 MGF technique, from slide 18 to slide 23. Question: Can we prove (⋆) strictly?
Remark: It is often easier to obtain the mgf than the cdf of the limiting dist..
4 Central limit theorem, from slide 24 to slide 36.
Proof and more applications.
5 ∆-method, from slide 37 to slide 44.
Question: Can we construct a (large-sample) confidence interval for 1/μ?
2/44
Boxiang Wang
Chapter 5 STAT 4101 Spring 2021
Definition
3/44
Motivation
In Chapter 4, we talked about the central limit theorem. For an iid sequence, X1, . . . , Xn, it holds that
√ n ( X ̄ n − μ ) Zn= σ
approximates N(0, 1) as n → ∞.
How can we formally define such approximation?
4/44
Boxiang Wang
Chapter 5 STAT 4101 Spring 2021
Definition: Convergence in Distribution
Let Fn and F be the cdf’s of random variables Xn and X. Let C denote the set of all points where F is continuous.
We say that Xn converges in distribution to X, denoted by Xn→D X,if
lim Fn(x) = F(x), ∀x ∈ C. n→∞
The distribution of X is called the asymptotic distribution or the limiting distribution of the sequence {Xn}.
When X has a specific distribution, say, N(2, 1), instead of writing “Xn →D X, where X ∼ N(2,1)”, we may abuse notation and write
Xn →D N(2,1).
5/44
Boxiang Wang
Chapter 5 STAT 4101 Spring 2021
Example (5.2.4)
Suppose X1, . . . Xn is a random sample from a uniform (0, θ) distribution. Let Yn = max(X1, . . . , Xn) and consider the random variable Zn = n(θ − Yn). Determine the limiting distribution of Zn.
Solution: Let t ∈ (0, nθ), the cdf of Zn is
P[Zn ≤t]=P[Yn ≥θ−(t/n)]
θ − (t/n)n =1− θ
t/θn =1−1−n
→1−e−t/θ, asn→∞,
where the last quantity is the cdf of an exponential distribution
random variable with mean θ, so we would say Zn →D exp(θ).
6/44
Boxiang Wang
Chapter 5 STAT 4101 Spring 2021
iid 1 n Suppose Xi ∼ N(0,1), i = 1,…,n. Let Xn = n i=1 Xi.
Determine the limiting distribution of {Xn} .
7/44
Boxiang Wang
Chapter 5 STAT 4101 Spring 2021
Example 5.2.2
Suppose {Xn} is a sequence of discrete random variables such that Xn has pmf
1 x=2+1 pn(x) = n
0 elsewhere. Find the limiting distribution of {Xn}.
8/44
Boxiang Wang
Chapter 5 STAT 4101 Spring 2021
Clearly, we see
lim pn(x) = 0, ∀x. n→∞
This shows that in general we can not determine the distribution of the limiting distribution of {Xn} by taking the limit of the pmf of Xn. In fact, we have to take the limit of the cdf to determine the limiting distribution.
ThecdfofXn is
Fn(x)= 1 x≥2+n−1 ,
0 x<2+n−1
0x≤2 lim Fn(x) = .
n→∞ 1 x>2 0x<2
Since
is a cdf and since limn→∞ Fn(x) = F(x) at all points of continuity of F(x), we say Xn →D X.
F(x)= 1 x≥2
9/44
Boxiang Wang
Chapter 5 STAT 4101 Spring 2021
Why Only Points of Continuity of FX ?
Consider a sequence of random variables {Xn, n = 1, 2, . . . } where Xn has a degenerate distribution at 1 , namely
P (Xn = 2 + 1 ) = 1. n
n
Let X be a random variable with all its mass at 2.
All the mass of Xn is converging to 0, the distribution of X.
At the point of discontinuity of FX ,
lim FXn(2)=0̸=1=FX(2). n→∞
At the point of continuity of FX ,
Hence Xn →D X.
lim FXn(x)=FX(x). n→∞
10/44
Boxiang Wang
Chapter 5 STAT 4101 Spring 2021
Example (5.2.3): Limiting Distribution of t
Let X1, . . . , Xn+1 be a random sample from N(0, 1). Then X1
n
Tn = n+1 X2 i=2 i
follows a t-distribution with n degrees of freedom. This is because n+1 X2 ∼ χ2 and n+1 X2 is independent of X .
i=2 i n i=2 i 1
An important consequence: when n is large, Tn ≈ X1 ∼ N(0, 1).
Why?
Using the law of large numbers,
n+1 X2
i=2 i ≈E[X2]=1, asn→∞,
so
X1 X1
Tn = ≈ n+1 X2
i=2 i n
1
=X1 ∼N(0,1), asn→∞.
n
11/44
Boxiang Wang
Chapter 5 STAT 4101 Spring 2021
Example (5.2.3): Limiting Distribution of t (cont’d)
Let Tn have a t-distribution with n degrees of freedom, n = 1,2,3,... Thus its cdf is
t
Fn(t) = √πnΓ(n/2)
−∞
(n+1)/2 dy. (1+y /n)
Γ[(n + 1)/2] 1 2
By the Lebesgue Dominated Convergence Theorem, we show
t t
lim Fn(t) = lim fn(y)dy = lim fn(y)dy.
n→∞ By showing
we have
n→∞ −∞ −∞n→∞
1 −y2/2 lim fn(y) = √ e
n→∞
,
lim Fn(t) = n→∞ −∞
√ e dy. 2π
2π
t 1−y2/2
Thus Tn has a limiting standard normal distribution.
12/44
Boxiang Wang
Chapter 5 STAT 4101 Spring 2021
Useful Results
13/44
Theorem 5.2.1. If Xn →P X, then Xn →D X.
Theorem 5.2.2. If Xn →D b, then Xn →P b, where b is a constant.
14/44
Boxiang Wang
Chapter 5 STAT 4101 Spring 2021
In general, Xn →D X does not imply that Xn →P X. Example:
Let X be a continuous random variable with a pdf fX (x) which is symmetric about 0, i.e., fX (x) = fX (−x).
It is easy to show that the density of the random variable −X is also fX (x).
Thus X and −X have the same distributions.
Define the sequence of random variable Xn as
X if n is odd Xn= −X ifniseven .
Clearly, FXn(x) = FX(x) for all x in the support of X, so that X n →D X .
On the other hand, the sequence Xn does not get close to X, say, Xn X in probability.
15/44
Boxiang Wang
Chapter 5 STAT 4101 Spring 2021
Theorem (5.2.4): Continuous Mapping Theorem
Suppose Xn →D X and g is a continuous function on the support of X. Then g(Xn) →D g(X).
The following theorem also holds.
Suppose Xn →P X and g is a continuous function on the support of X. Then g(Xn) →P g(X).
16/44
Boxiang Wang
Chapter 5 STAT 4101 Spring 2021
Slutsky’s Theorem
IfXn →D X andYn →P awhereaisaconstant. Then 1 XnYn→DaX;
2 X n + Y n →D X + a .
Example: Let X1,...,Xn ∼ (μ,σ2), then by CLT, X ̄ n − μ D
Zn = σ/√n −→N(0, 1).
It is known that the sample standard deviation S is a consistent
estimator of σ, i.e., S →P σ. Thus by Slutsky’s theorem,
X ̄n−μ σ D
Tn= S/√n =S·Zn−→N(0,1).
17/44
Boxiang Wang
Chapter 5 STAT 4101 Spring 2021
MGF Technique
18/44
Recall the definition of convergence in distribution.
We say that Xn converges in distribution to X, denoted by Xn→D X,if
lim Fn(x) = F(x), ∀x ∈ C. n→∞
What if the cdf FXn (x) is hard to obtain in closed form?
If the mgf exists, can it provide a convenient method of determining the limiting distribution?
19/44
Boxiang Wang
Chapter 5 STAT 4101 Spring 2021
Theorem 5.2.10
Let {Xn} be a sequence of random variables with mgf MXn (t) that exists for |t| < h for all n. Let X be a random variable with mgf
M (t) which exists for |t| < h.
If
then
lim MXn(t) = M(t) for all |t| < h, n→∞
X n →D X .
20/44
Boxiang Wang
Chapter 5 STAT 4101 Spring 2021
Example 5.2.6
Yn ∼ Bin(n, p) such that the mean np = μ is the same for every n. Then what is the limiting distribution of Yn?
Solution:
We see the limit of MYn (t): MYn(t)=EetYn=(1−p)+petn = 1+μ e −1
t n n
for all real values of t. Hence we have
lim MYn (t) = eμ(et−1),
n→∞
which indicates that Yn has a limiting Poisson distribution with mean μ.
Another solution will be seen in slide 32.
21/44
Boxiang Wang
Chapter 5 STAT 4101 Spring 2021
Exercise 5.2.7
Let Xn have a gamma distribution with parameter α = n and β, where β is not a function of n. Let Yn = Xn/n. Find the limiting distribution of Yn.
22/44
Boxiang Wang
Chapter 5 STAT 4101 Spring 2021
Example 5.2.7
Let Zn be χ2(n) and let Yn = (Zn − n)/√2n. Show that the limiting distribution of Yn is a standard normal distribution. Another solution based on the CLT will be seen in slide 36.
23/44
Boxiang Wang
Chapter 5 STAT 4101 Spring 2021
Chapter 5
Consistency and Limiting Distributions
5.3 Central Limit Theorem
24/44
Boxiang Wang
Chapter 5 STAT 4101 Spring 2021
Central Limit Theorem
Let X1, . . . , Xn denote the observations of a random sample from a distribution that has mean μ and positive variance σ2. Then the
random variable
X ̄ n − μ D
Yn= σ/√n →N(0,1).
Remark: In the proof, we assume that the mgf M (t) = E(etX ) exists for −h < t < h. Use characteristic function without this assumption.
In other words, as n is sufficiently large,
n
Xi is approximately distributed as N(nμ, nσ2);
i=1
̄ σ2
Xn is approximately distributed as N μ, n .
25/44
Boxiang Wang
Chapter 5 STAT 4101 Spring 2021
Examples of CLT
Example1.IfX1,...,Xn areiidBern(p):μ=p,σ2 =p(1−p), √ n ( X ̄ − p ) D
p(1−p) →N(0,1).
Example 2. If X1,...,Xn are iid Pois(λ) : μ = λ,σ2 = λ,
√ n ( X ̄ − λ ) D
√
→ N(0, 1).
Example 3. If X1,...,Xn are iid Unif(0,θ) : μ = θ, σ2 = θ2 , 2 12
λ
√ n X ̄ − θ D 2
θ2 12
→ N(0, 1).
26/44
Boxiang Wang
Chapter 5 STAT 4101 Spring 2021
Proof of CLT
X ̄ n − μ
Define Yn = σ/√n . Our goal is to show that
lim MYn (t) = et2/2. n→∞
For ease of presentation, we define Ui = Xi−μ. √σ
We observe that Yn = nU ̄n, and E(U1) = 0 and Var(U1) = 1.
We have
MU1 (t) = E exp(tU1).
M (0)=1,M′ (0)=0, andM′′ (0)=1. U1 U1 U1
27/44
Boxiang Wang
Chapter 5 STAT 4101 Spring 2021
MYn (t) = E [exp (tYn)]
= E exp √ntU ̄n
t n =Eexp√n Ui
i=1
t t
t =E exp √nU1 exp √nU2 ...exp √nUn
t t =E exp √nU1 E exp √nU2
t n = E exp √nU1
tn t
= MU1 √n , −h < σ√n < h.
t ...E exp √nUn
Once again our goal is
lim MYn (t) = et2/2. n→∞
28/44
Boxiang Wang
Chapter 5 STAT 4101 Spring 2021
log lim MYn(t) = lim logMYn(t) = lim nlogMU1 n→∞ n→∞ n→∞
Letz= 1 ,thenn= 1 ,andthus √n z2
lim log MYn (t) = lim log MU1 (zt)
n→∞
√
n
.
t
z→0 z2
t M′ (zt) = lim U1
2 z→0 zMU1 (zt)
t M′′ (zt)t =lim U1
which gives
t M′′ (0)t = U1
2MU1(0)+0M′′ (0) U1
t1·t t2 =21+0·1= 2.
lim MYn (t) = et2/2, n→∞
2z→0MU1(zt)+ztM′′ (zt) U1
and thus Yn →D N(0, 1).
Boxiang Wang
Chapter 5 STAT 4101 Spring 2021
29/44
Demonstration of CLT
The animation can be found on http://onlinestatbook.com/stat_sim/sampling_dist/.
30/44
Boxiang Wang
Chapter 5 STAT 4101 Spring 2021
Example (5.3.2)
Let X denote the mean of a random sample of size 75 from the distribution with pdf
1 0