University of California, Los Angeles Department of Statistics
Instructor: Nicolas Christou
Exponential families
A probability density function or probability mass function is called an exponential family if it can be expressed as
k f(x|θ) = h(x)c(θ)exp wi(θ)ti(x) .
i=1
Note: h(x), t1(x), . . . , tk(x) do not depend on θ and c(θ) does not depend of x.
Example:
Consider X ∼ b(n, p) with n fixed. Show that p(x) = npx(1 − p)n−x can be expressed in
Statistics 100B
the exponential family form.
n
x px(1−p)n−x
n p x
x 1−p (1−p)n
p(x) = = = =
h(x)=n,c(p)=(1−p)n,t1(x)=x,w1(p)=log p . x 1−p
x
n px
(1−p)nelog(1−p) n p
x x
(1 − p)nexlog( 1−p )
Therefore this pmf is an exponential family with
Theorem:
Suppose a random variable X has a pdf or pmf that can be expressed in the form of expo- nential family. Then,
k∂wi(θ) ∂
(a) E ti(x) =− logc(θ).
and
i=1 ∂θj ∂θj k ∂wi(θ) ∂2
(b) var i=1
k ∂2wi(θ) ti(x) =− logc(θ)−E ti(x) .
∂θj
Note: Here log is the natural logarithm.
∂θj2
i=1
∂θj2
1
Proof of (a):
x k
h(x)c(θ)exp wi(θ)ti(x) i=1
dx
x
Differentiate both sides w.r.t. θj:
∂c(θ)k
f (x|θ)dx
= 1 = 1
exp wi(θ)ti(x) dx
h(x)
k∂wi(θ) k
∂θj
+ h(x)c(θ) ti(x)exp wi(θ)ti(x) dx = 0
x
i=1
x i=1 ∂θj i=1
Multiply the first integral by c(θ) and note that ∂logc(θ) = ∂c(θ) 1 .
∂c(θ) ∂θj
c(θ) ∂θj k c(θ)
ti(x)exp
∂θj c(θ)
dx = 0
wi(θ)ti(x) dx i=1 c(θ)
exp
k∂wi(θ) k
h(x)
+ h(x)c(θ)
k ∂wi(θ)
x
x i=1 ∂θj After rearranging we get
wi(θ)ti(x) i=1
ti(x)h(x)c(θ)exp
∂logc(θ) k
k
wi(θ)ti(x) dx =
x i=1
E
∂θj
− h(x)c(θ)exp
∂θj
ti(x) = − i=1 ∂θj
i=1
logc(θ).
Or
∂θj x k∂wi(θ) ∂
wi(θ)ti(x) dx i=1
To prove statement (b) of the theorem differentiate a second time and rearrange.
2