Chapter 6 Maximum Likelihood Methods
6.1 Maximum Likelihood Estimation
1/21
Boxiang Wang
Chapter 6 STAT 4101 Spring 2021
Outline
1 Definition of MLE
2 Examples of MLE
3 Properties of MLE
2/21
Boxiang Wang
Chapter 6 STAT 4101 Spring 2021
Definition of MLE
3/21
Motivation
A simple example:
Let (X1, X2) be a random sample of size 2 from Bin(10, p),
wherep=1 or1,i.e.,Ω={1, 1}.Supposethat 2323
X1 =5,X2 =6.
Which value is more likely for the true p, 1 or 1 ? 23
4/21
Boxiang Wang
Chapter 6 STAT 4101 Spring 2021
Likelihood Function
Let f (X; θ) denote the joint pdf or pmf of the sample
X = (X1,··· ,Xn). Then for fixed X, the function of θ, defined by
L(θ;X)=f(X; θ)
is called the likelihood function. Remark:
1 L(θ; X) is a function in terms of θ. We use L(θ) or L(θ; X) interchangeably for simplicity.
2 If X1,··· ,Xn is a random sample from f(x), then the likelihood function is
n
L(θ; X) = f(xi;θ), θ ∈ Ω.
i=1
3 If the distribution is discrete, the likelihood function gives the probability of observing the given observed data
X1 = x1,··· ,Xn = xn as a function of θ.
5/21
Boxiang Wang
Chapter 6 STAT 4101 Spring 2021
Maximum Likelihood Estimate
Given observed data X = (x1, · · · , xn), the maximum likelihood estimate (mle) of θ is a point in the parameter space Ω at which L(θ; x) attains its maximum.
Remark:
1 We use the symbol θˆ = θˆ(x) to denote an mle of θ: θˆ(x) = arg max L(θ; x).
θ
2 Note that θ maximizes L(θ; X) if and only if θ maximizes l(θ; X) := log(L(θ; X)).
3 l(θ; X) is sometimes easier to work with mathematically because
n
l(θ;x) = logL(θ;x) = logf(xi;θ).
i=1
6/21
Boxiang Wang
Chapter 6 STAT 4101 Spring 2021
In this course we only consider the cases where L(θ) has a unique maximizer.
Therefore the MLE is uniquely defined. Hence for any
t ̸= θˆMLE,
We can solve θˆ
or
L(θˆMLE) > L(t). by solving
∂L(θ) = 0 ∂θ
∂l(θ) = 0 ∂θ
MLE
In fact, we need to verify if θˆ is maximum. We may check MLE
this by showing
∂2L(θ) < 0. ∂2θ
In class, we only consider the problems where θˆMLE is the unique root to the problems above and we don’t check the 2nd derivative.
7/21
Boxiang Wang
Chapter 6 STAT 4101 Spring 2021
Examples of MLE
8/21
Example (4.1.2): MLE for Bernoulli Distribution
Let X1, · · · , Xn be a random sample from the distribution with pmf p(x;θ)=θx(1−θ)1−x forθ∈[0,1].Findthemleofθ.
Solution:
Step 1: Write down the likelihood and log-likelihood function. n
L(θ)=p(xi;θ)=θni=1xi(1−θ)n−ni=1xi, i=1
n n l(θ)=xilogθ+ n−xi log(1−θ),
xi =0or1, xi =0or1.
i=1 ∂l(θ)
ni=1 xi θ
i=1
n − ni=1 xi
Step2: ∂θ = ni=1 xi
−
− 1−θ . n − ni=1 xi
1 − θ = 0.
Step 3: θ
Step 4: solve θMLE = X ̄ , which is unbiased in this example.
9/21
Boxiang Wang
Chapter 6 STAT 4101 Spring 2021
10/21
Boxiang Wang
Chapter 6 STAT 4101 Spring 2021
Example (4.1.1): MLE for Exponential Distribution
Let X1, · · · , Xn denote a random sample from the distribution with pdf
1e−x/θ f(x;θ) = θ
θ e i Step2: ∂θ =−nθ +θ
i=1 n
Step 1: l(θ) = log
∂l(θ) −1 −2n
0
Remark. This theorem says, asymptotically, the true value θ0 maximizes the likelihood function.
17/21
Boxiang Wang
Chapter 6 STAT 4101 Spring 2021
Theorem (6.1.2): Invariance Property of MLE
Let θˆMLE be the mle of θ and g(θ) is a function of θ. Then the mle of g(θ) is g(θˆMLE).
Example:
Example:
g(θ) = θ2, then its mle is θˆ2 , MLE
g(θ) = 1/θ, then its mle is 1/θˆMLE, g(θ) = exp(θ), then its mle is exp(θˆMLE).
Let X1,…,Xn ∼ Bern(p). We have seen that pˆMLE = X ̄, based on the invariance property, we also see that the mle of p(1 − p) is X ̄(1 − X ̄).
18/21
Boxiang Wang
Chapter 6 STAT 4101 Spring 2021
Example
1−x Let X1,…,Xn be a random sample from f(x;θ) = θe θ .
Compute the mle of η = 1/θ.
19/21
Boxiang Wang
Chapter 6 STAT 4101 Spring 2021
Example
Poisson distribution is often used to model rare events. Suppose
X represents the number of the rare events. We assume X follows
Poi(λ) and we are interested in the probability Pr(X = 0), that is,
the chance that the rare event won’t occur.
To estimate this probability, we collected 100 samples from the
distribution Poi(λ) and we observed 100 xi = 29. Use this i=1
information to find the MLE of Pr(X = 0).
Solution: The log-likelihood of Poisson distribution is
n n1 l(λ)=−nλ+ xi log(λ)+log ,
Since
P(λ) ≡ Pr(X = 0) = e−λ,
by the invariance principle, we conclude that the MLE of Pr(X = 0) is e−0.29 = 0.748.
i=1 i=1 xi! which implies that λˆmle = X ̄. In this problem λˆmle = 0.29.
20/21
Boxiang Wang
Chapter 6 STAT 4101 Spring 2021
Corollary (6.1.1): Consistency of MLE
Assume that X1, · · · , Xn satisfy the following regularity conditions:
(R0) The pdfs are distinct; i.e., θ ̸= θ′ ⇒ f(x; θ) ̸= f(x; θ′). (R1) The pdfs have common support for all θ.
(R2) The true parameter θ0 is an interior point in Ω.
Suppose that f (x; θ) is differentiable with respect to θ ∈ Ω. and the likelihood equation has the unique solution θˆn. Then
θ ˆ n →P θ 0 .
21/21
Boxiang Wang
Chapter 6 STAT 4101 Spring 2021