CS计算机代考程序代写 Chapter 6 Maximum Likelihood Methods

Chapter 6 Maximum Likelihood Methods
6.1 Maximum Likelihood Estimation
1/21
Boxiang Wang
Chapter 6 STAT 4101 Spring 2021

Outline
1 Definition of MLE
2 Examples of MLE
3 Properties of MLE
2/21
Boxiang Wang
Chapter 6 STAT 4101 Spring 2021

Definition of MLE
3/21

Motivation
A simple example:
􏰉 Let (X1, X2) be a random sample of size 2 from Bin(10, p),
wherep=1 or1,i.e.,Ω={1, 1}.Supposethat 2323
X1 =5,X2 =6.
􏰉 Which value is more likely for the true p, 1 or 1 ? 23
4/21
Boxiang Wang
Chapter 6 STAT 4101 Spring 2021

Likelihood Function
Let f (X; θ) denote the joint pdf or pmf of the sample
X = (X1,··· ,Xn). Then for fixed X, the function of θ, defined by
L(θ;X)=f(X; θ)
is called the likelihood function. Remark:
1 L(θ; X) is a function in terms of θ. We use L(θ) or L(θ; X) interchangeably for simplicity.
2 If X1,··· ,Xn is a random sample from f(x), then the likelihood function is
n
L(θ; X) = 􏰅f(xi;θ), θ ∈ Ω.
i=1
3 If the distribution is discrete, the likelihood function gives the probability of observing the given observed data
X1 = x1,··· ,Xn = xn as a function of θ.
5/21
Boxiang Wang
Chapter 6 STAT 4101 Spring 2021

Maximum Likelihood Estimate
Given observed data X = (x1, · · · , xn), the maximum likelihood estimate (mle) of θ is a point in the parameter space Ω at which L(θ; x) attains its maximum.
Remark:
1 We use the symbol θˆ = θˆ(x) to denote an mle of θ: θˆ(x) = arg max L(θ; x).
θ
2 Note that θ maximizes L(θ; X) if and only if θ maximizes l(θ; X) := log(L(θ; X)).
3 l(θ; X) is sometimes easier to work with mathematically because
n
l(θ;x) = logL(θ;x) = 􏰄logf(xi;θ).
i=1
6/21
Boxiang Wang
Chapter 6 STAT 4101 Spring 2021

􏰉 In this course we only consider the cases where L(θ) has a unique maximizer.
􏰉 Therefore the MLE is uniquely defined. Hence for any
t ̸= θˆMLE,
􏰉 We can solve θˆ
or
L(θˆMLE) > L(t). by solving
∂L(θ) = 0 ∂θ
∂l(θ) = 0 ∂θ
MLE
􏰉 In fact, we need to verify if θˆ is maximum. We may check MLE
this by showing
∂2L(θ) < 0. ∂2θ In class, we only consider the problems where θˆMLE is the unique root to the problems above and we don’t check the 2nd derivative. 7/21 Boxiang Wang Chapter 6 STAT 4101 Spring 2021 Examples of MLE 8/21 Example (4.1.2): MLE for Bernoulli Distribution Let X1, · · · , Xn be a random sample from the distribution with pmf p(x;θ)=θx(1−θ)1−x forθ∈[0,1].Findthemleofθ. Solution: Step 1: Write down the likelihood and log-likelihood function. n L(θ)=􏰅p(xi;θ)=θ􏰃ni=1xi(1−θ)n−􏰃ni=1xi, i=1 n 􏰐n􏰑 l(θ)=􏰄xilogθ+ n−􏰄xi log(1−θ), xi =0or1, xi =0or1. i=1 ∂l(θ) 􏰃ni=1 xi θ i=1 n − 􏰃ni=1 xi Step2: ∂θ = 􏰃ni=1 xi − − 1−θ . n − 􏰃ni=1 xi 1 − θ = 0. Step 3: θ Step 4: solve θ􏰛MLE = X ̄ , which is unbiased in this example. 9/21 Boxiang Wang Chapter 6 STAT 4101 Spring 2021 10/21 Boxiang Wang Chapter 6 STAT 4101 Spring 2021 Example (4.1.1): MLE for Exponential Distribution Let X1, · · · , Xn denote a random sample from the distribution with pdf 􏰂 1e−x/θ f(x;θ) = θ θ e i Step2: ∂θ =−nθ +θ i=1 n Step 1: l(θ) = log ∂l(θ) −1 −2􏰄n 0 L(θ,X)] = 1, for all θ ̸= θ0. n→∞
Remark. This theorem says, asymptotically, the true value θ0 maximizes the likelihood function.
17/21
Boxiang Wang
Chapter 6 STAT 4101 Spring 2021

Theorem (6.1.2): Invariance Property of MLE
Let θˆMLE be the mle of θ and g(θ) is a function of θ. Then the mle of g(θ) is g(θˆMLE).
Example:
Example:
g(θ) = θ2, then its mle is θˆ2 , MLE
g(θ) = 1/θ, then its mle is 1/θˆMLE, g(θ) = exp(θ), then its mle is exp(θˆMLE).
Let X1,…,Xn ∼ Bern(p). We have seen that pˆMLE = X ̄, based on the invariance property, we also see that the mle of 􏰓p(1 − p) is 􏰓X ̄(1 − X ̄).
18/21
Boxiang Wang
Chapter 6 STAT 4101 Spring 2021

Example
1−x Let X1,…,Xn be a random sample from f(x;θ) = θe θ .
Compute the mle of η = 1/θ.
19/21
Boxiang Wang
Chapter 6 STAT 4101 Spring 2021

Example
Poisson distribution is often used to model rare events. Suppose
X represents the number of the rare events. We assume X follows
Poi(λ) and we are interested in the probability Pr(X = 0), that is,
the chance that the rare event won’t occur.
To estimate this probability, we collected 100 samples from the
distribution Poi(λ) and we observed 􏰃100 xi = 29. Use this i=1
information to find the MLE of Pr(X = 0).
Solution: The log-likelihood of Poisson distribution is
􏰐n􏰑 􏰐n1􏰑 l(λ)=−nλ+ 􏰄xi log(λ)+log 􏰅 ,
Since
P(λ) ≡ Pr(X = 0) = e−λ,
by the invariance principle, we conclude that the MLE of Pr(X = 0) is e−0.29 = 0.748.
i=1 i=1 xi! which implies that λˆmle = X ̄. In this problem λˆmle = 0.29.
20/21
Boxiang Wang
Chapter 6 STAT 4101 Spring 2021

Corollary (6.1.1): Consistency of MLE
Assume that X1, · · · , Xn satisfy the following regularity conditions:
(R0) The pdfs are distinct; i.e., θ ̸= θ′ ⇒ f(x; θ) ̸= f(x; θ′). (R1) The pdfs have common support for all θ.
(R2) The true parameter θ0 is an interior point in Ω.
Suppose that f (x; θ) is differentiable with respect to θ ∈ Ω. and the likelihood equation has the unique solution θˆn. Then
θ ˆ n →P θ 0 .
21/21
Boxiang Wang
Chapter 6 STAT 4101 Spring 2021