University of California, Los Angeles Department of Statistics
Statistics 100B Instructor: Nicolas Christou
Multivariate normal distribution
One of the most important distributions in statistical inference is the multivariate normal distribution. The probability density function of the multivariate normal distribution, its moment generating function, and its properties are discussed here.
Probability density function
We say that a random vector Y = (Y1, Y2, . . . , Yn)′ with mean vector μ and variance covari- ance matrix Σ follows the multivariate normal distribution if its probability density function is given by
1 −1 −1(Y−μ)′Σ−1(Y−μ)
f(Y)= n |Σ| 2e 2 , (1)
(2π)2
and we write, Y ∼ Nn(μ, Σ). If Y = (Y1, Y2) then we have a bivariate normal distribution
and its probability density function can be expressed as
1 f(y1,y2) = 2πσσ√1−ρ2
12
inverse of Σ and substituting it into (1). Here is the bivariate normal pdf. Bivariate Normal Distribution
0.015
0.010
0.005
0.000 −10
10 5
1
× exp −2(1−ρ2)
y1 −μ1y2 −μ2 −2ρ σ σ
1212 Here, we have Σ = 1 12 . The previous expression can be obtained by finding the
σ2 σ σ21 σ2
f(x,y)
−5
0
1
y1 −μ12 y2 −μ22 σ + σ
y
0 −5 5
10 −10
x
Moment generating function
A useful tool in statistical theory is the moment generating function. The joint moment generating function is defined as MY(t) = Eet′Y = Eeni=1 yiti ,, where Y = (Y1, Y2, . . . , Yn)′ and t = (t1, t2, . . . , tn)′. We will find the joint moment generating function of Y ∼ Nn(μ, Σ).
• Suppose Z ∼ Nn(0, I). Since Z1, Z2, . . . , Zn are independent the joint moment gener-
ating function of Z is M (t) = e1 t′t. Z2
n2
• IfY∼N (μ,Σ)showthatZ=Σ−1(Y−μ)followsN(0,I).
• Aside note: What is Σ− 1 ?
2
• To find the joint moment generating function of Y ∼ Nn(μ,Σ) we use the transformation 2Y2
Y=Σ1Z+μtogetM (t)=et′μ+1t′Σt.
3
Theorem 1
Let Y ∼ Nn(μ,Σ), and let A be an m×n matrix of rank m and c be an m×1 vector. Then AY + c ∼ Nm(Aμ + c, AΣA′).
Proof
4
Theorem 2
Let Y ∼ Nn(μ, Σ). Sub-vectors of Y follow the multivariate normal distribution and linear combinationsofY1,Y2,…,Yn followtheunivariatenormaldistribution.
Proof
Suppose Y, μ, and Σ are partitioned as follows Y = 1 , μ = 1 , Σ = 11 12 ,
ofY ,Y ,…,Y ,i.e. a Y +a Y +…+a Y =a′Y,thematrixAoftheorem1isavector 12n1122nn
Q2 μ2 Σ21 Σ22 where Y1 is p×1. We will show that Q1 ∼ Np(μ1, Σ11) and Q2 ∼ Nn−p(μ2, Σ22). The result follows directly by using the previous theorem with A = (Ip,0). For a linear combination
and therefore, a′Y ∼ N(a′μ,
Example
√
a′Σa).
σ12 σ12 σ13 σ14 σ15
Q μ ΣΣ
Y1 μ1
Y2 μ2
σ21 σ2 σ23 σ24 σ25
Y LetY=Y ,μ=μ ,Σ=σ σ σ2 σ σ ,thenifQ = 1 ,
3 3 31 32 3 34 35 1 Y 2 2 Y4 μ4 σ41 σ42 σ43 σ4 σ45
Y5 μ5 σ51 σ52 σ53 σ54 σ52
.
itfollowsthatQ ∼N
μ σ2 σ 1 , 1 12
1 μ2 σ21σ2
5
Statistical independence
Suppose Y, μ, Σ are partitioned as in theorem 2. We say that Q1, Q2 are statistically inde-
pendent if and only if Σ12 = 0. We can show this using the joint moment generating function
of Y. Recall that the exponent of the joint moment generating function of the multivariate
normal distribution is t′ μ + 1 t′ Σt which after partitioning t conformably (according to the 2
partitioning of Y,μ,Σ) can be expressed as t1′μ1+t2′μ2+1t1′Σ11t1+1t2′Σ22t2+t1′Σ12t2. 22
When Σ12 = 0, the joint moment generating function can be expressed as the product of the two marginal moment generating functions of Q1 and Q2, i.e. MY(t) = MQ1 (t1)MQ2 (t2), therefore, Q1 and Q2 are independent.
Theorem 3
Using theorem 1 and the statement about statistical independence above, we prove the fol- lowing theorem. Suppose Y ∼ Nn(μ,Σ) and define the following two vectors W1 = AY and W2 = BY. Then, W1 and W2 are independent if cov(W1, W2) = AΣB′ = 0.
Y = LY. Therefore using
Proof
We stack the two vectors as follows: W = 1 =
theorem 1 we find that W ∼ N(Lμ,LΣL′) or A AΣA′ AΣB′
WA W2 B
W ∼ N B μ, BΣA′ BΣB′ , and we conclude that W1 and W2 are independent if and only if AΣB′ = 0.
Conditional probability density functions for multivariate normal
Consider the bivariate normal distribution (see page 1). From theorem 1 it follows that Y1 ∼ N(μ1,σ1). This is also called the marginal probability distribution of Y1. We want to find the conditional distribution of Y2 given Y1.
From the conditional probability law, fY2|Y1(y2|y1) = fY1Y2(y1,y2), and after substituting the fY1 (y1 )
bivariate density and the marginal density it can be shown that the conditional probability density function of Y2 given Y1 is given by
1 σ2(1 − ρ)2
1Y2 −μ2 −ρσ2(Y1 −μ1) σ1
fY2|Y1(y2|y1)=
We recognize that this is a normal probability density function with mean
μY2|Y1 = μ2 + ρσ2 (Y1 − μ1) and variance σ2 = σ2(1 − ρ2). σ1 Y2 |Y1
6
√ exp −
2π 2 σ2(1 − ρ2)
.
In general:
Suppose that Y, μ, and Σ are partitioned as follows Y = 1 , μ = 1 , Σ = 11 12 ,
and Y ∼ N(μ,Σ). It can be shown that the conditional distribution of Q1 given Q2 is also mul-
tivariate normal, Q1|Q2 ∼ N(μ1|2,Σ1|2), where μ1|2 = μ1 + Σ12Σ−1(Q2 − μ2), and Σ1|2 = 22
Σ11 − Σ12Σ−1Σ21. 22
Proof:
LetU=Q1−Σ12Σ−1Q2 andV=Q2. 22
7
QμΣΣ
Q2 μ2 Σ21 Σ22
Example 1
Suppose the prices (in $), Y1,Y2,Y3,Y4 of objects A, B, C, and D are jointly normally distributed as Y1 1 3 2 3 3
Y2 3 2 5 5 4 Y ∼ N4(μ,Σ), where, Y = , μ = , and Σ = .
Y3 6 3 5 9 5 Y4 4 3456
Answer the following questions:
a. Suppose a person wants to buy three of product A, four of product B, and one of product C. Find the probability that the person will spend more than $30.
b. Find the moment generating function of Y1.
c. Find the joint moment generating function of (Y1,Y3).
d. Find the correlation coefficient between Y3 and Y4.
Example 2
Y1 2 2 1 1
SupposeY∼N3(μ,Σ),whereY= Y2 ,μ= 1 ,andΣ= 1 3 0 . Findthejoint Y3 2 101
distributionofQ1 =Y1+Y2+Y3 andQ2 =Y1−Y2. Example 3
Answer the following questions:
1
1
a. Let X ∼ Nn(μ1,Σ), where 1 = . , and Σ is the variance covariance matrix of X. Let
. 1
1 n−1
1000 0 1 0 0
b. Suppose ε ∼ N3(0,σ2I3) and that Y0 ∼ N(0,σ2), independently of the εi’s. Therefore Y0
ε1
the vector , is multivariate normal. Define Yi = ρYi−1 + εi for i = 1, 2, 3. Express
Y1, Y2, Y3 in terms of ρ, Y0, and the εi’s.
Y1
c. Refer to part (b). Find the covariance matrix of Y = Y2 ,
Σ = (1 − ρ)I + ρJ, with ρ > −
Therefore, when ρ = 0 we have X ∼ Nn(μ1,I), and in this case we showed in class that X ̄
ε2 ε3
d. What is the distribution of Y?
, I =
. . . . and J = ….
.. ..
. . . . . …. .. ..
1111 1 1 1 1
0001 1111 and ni=1(Xi − X ̄)2 are independent. Are they independent when ρ ̸= 0?
8
Y3