Probabilities and Expectations
Ashique Rupam Mahmood September 9, 2015
Probabilities tell us about the likelihood of an event in numbers. If an event is certain to occur, such as sunrise, probability of that event is said to be 1. Pr(sunrise) = 1. If an event will certainly not occur, then its probability is 0.
So, probability maps events to a number in [0,1]. How do you specify an event? In the discussions of probabilities, events are technically described as a set. At this point it is important to go through some basic concepts of sets and maybe also functions.
Sets
A set is a collection of distinct objects. For example, if we toss a coin once, the set of all possible distinct outcomes will be S = {head,tail}, where head denotes a head and the tail denotes a tail. All sets we consider here are finite.
An element of a set is denoted as head ∈ S. A subset of a set is denoted as {head} ⊂ S. What are the possible subsets of S? These are: {head},{tail},S = {head,tail}, and φ = {}. So, note that a set is a subset of itself: S ⊂ S. Also note that, an empty set (a collection of nothing) is a subset of any set: φ ⊂ S. A union of two sets A and B is comprised of all the elements of both sets and denoted as A ∪ B. An intersection of two sets A and B is comprised of only the common elements of both sets and denoted as A ∩ B. A complement set of A in B is a set comprising the elements of B that are not in A and denoted as B \ A or B − A. The Cartesian product of two sets A and B is a set denoted as A × B comprising all ordered pairs (a, b) where a ∈ A and b ∈ B.
More about Sets (can be skipped)
ApowersetofasetSisasetofallthesubsetsofS. Itisoftendenotedas2S. Inour example, it is 2S = {{head} , {tail} , S, φ}. Therefore, S ∈ 2S . How many distinct elements we have in 2S? It is 2n, if n is the number of distinct elements in S.
1
Functions
A function is a map from a one set to another. A function takes an argument and associates it with exactly one quantity, which is often called the output of a function. The set of all the arguments a function takes is called the domain. The set of all the outputs that a function associates with is called a codomain, sometimes also known as range.
For example, if we want to associates the outcomes of a coin toss, head and tail, with numbers 1 and 0, we can do so by using functions. We can say, f(head) = 1 and f(tail) = −1. Then the domain of the function f is S = {head,tail} and the codomain is V = {1, −1}. This function can also be denoted as f : S → V .
Now, note that if a relation f associates more than one output with an argument, e.g., f(head) = 1 or f(head) = 0, then f is not a function. However, a function can map two different arguments to a single output. Hence, if f(head) = 1 and f(tail) = 1, then f is a function.
Sample Space, Events & Probabilities
Let us assume there is an experiment that is repeatable, such as rolling a dice. There are two important elements that make up a probabilistic model: a sample space and a probability distribution.
A sample space is the set of all possible outcomes of an experiment. Therefore, in the dice-rolling experiment, the sample space S is {1, 2, 3, 4, 5, 6}. An event is any subset of the sample space. For example, the event that a number more than two would appear in the dice-rolling experiment is {3, 4, 5, 6}. It makes sense to find the probability of such an event. An event can comprise only one outcome, for example, the event that 2 will appear: {2} or simply 2. A complementary event of an event A is Ac = S − A.
Now, we are ready to talk about probabilities. A probability is a function that associates an event with a non negative number. Therefore, for any event A
Pr(A) ≥ 0. (1) A probability function has certain key properties. For example, the addition of probabilities
of all the outcomes is always 1:
Pr(e) = 1. (2) e∈S
How the probability is distributed among the outcomes is defined by the probability distri- bution. For example, for the experiment of rolling an unbiased dice, the probability distri-
2
bution can be uniform, that is, it is the same for all the outcomes: for each outcome e ∈ S, Pr(e) = 1/6.
There are other properties of a probability function. For example, if two events A and B are disjoint (no common elements), then the probability of their union is the addition of their separate probabilities:
Pr(A∪B)=Pr(A)+Pr(B); A∩B=φ. (3)
The outcomes of an experiment are always mutually disjoint.
So, we talked about three properties of a probability function: nonnegativity (equation
1), normalization (equation 2) and additivity (equation 3).
If we are given to find the probability of an event, one way would be to find the sample
space and the probability distribution among the outcomes. Then we need to specify the event in question. Then using the additive rule it becomes easy to find the probability of that event.
Now, we are ready to calculate the probability that a number more than 2 will appear in the unbiased dice rolling experiment. Using the properties in the above, we can write:
Pr({3, 4, 5, 6}) = 4×1/6=2/3.
Pr(observe a number more than 2) =
= Pr(3) + Pr(4) + Pr(5) + Pr(6); additivity
More about Events & Probabilities (Can be skipped)
How many distinct events are possible in a dice rolling experiment? It is the number of all possible subsets of the sample space. The set of all possible distinct events is the power set of the sample space. The empty set φ is a possible event. An empty-set event is the event where nothing appears in a dice rolling experiment. Probability of an empty-set event is zero. This is an event that will certainly not occur. The sample space itself is also an event. It denotes the event that either of the possible outcomes would appear in the dice rolling experiment. This event will certainly occur. Probability of the sample space is always one: Pr(S) = 1.
So, now we understand that probability is in fact a function that takes an event as the argument and gives a number between 0 and 1, inclusively, as an output. We can write Pr:2S →[0,1].
3
Random Variables
It is often more convenient to express events in relations with random variables. A random variable takes the possible values of the outcomes of an experiment. In the dice-rolling experiment, a random variable X would be such that X ∈ {1, 2, 3, 4, 5, 6}.
Wecannowwritetheeventofhavinganumbermorethan2asX>2. Here,X>2 stands for the event {3, 4, 5, 6}. For the events with only one outcome, we usually write as X =2orX =3orX =xandsoon. Whenweuseletters,randomvariableswillbedenoted by capital letters and values of a random variable will be denoted by small letters.
Examples of Probabilities
In the dice rolling example, if the random variable X stands for the outcome, what is Pr(X > 1)? Now,
Pr(1≤X≤6) Pr([X =1]∪[2≤X ≤6]) Pr([X = 1]∪[X > 1]) Pr(X = 1)+Pr(X > 1) Pr(X > 1) Pr(X > 1)
Conditional Probabilities
Probability of an event A given another event B is defined as Pr(A|B)=. Pr(A∩B).
Here =. stands for equality by definition rather than by derivation. Pr(A∩B) is also written as Pr(A, B) or Pr(A&B).
When an event is already given or known to have happened, the uncertainty of another event might change. For example, in the dice rolling experiment, Pr(3) is 1/6. However, if we already know that the outcome is an odd number, does the event of observing 3 remains uncertain in the same amount? Actually not. It is now more likely to see 3 than before. Conditional probability reflects such a case. It can be calculated in the following way:
Pr(X = 3|X is odd) = Pr(X = 3 ∩ X is odd) = Pr(3 ∩ {1, 3, 5}) = Pr(3) = 1/6 = 1 . Pr(X is odd) Pr({1, 3, 5}) Pr({1, 3, 5}) 3/6 3
= 1
= 1
= 1
= 1
= 1 − Pr(X = 1) = 5/6.
Pr(B)
4
The uncertainty of observing 3 has indeed changed. It has become more likely to be observed. Another example. What is the probability to observe an odd number, when we know
that it is a multiple of 3?
Pr(X is odd|X is a multiple of 3) = Pr(X is odd ∩ X is a multiple of 3) =
Pr( is a multiple of 3) Pr({1,3,5}∩{3,6}) = Pr(3) = 1/6 = 1.
Pr({3, 6}) Pr({3, 6}) 2/6 2
When the likelihood of an event does not change after knowing another event, then those
two events are said to be independent:
[A is independent of B] ⇔ [Pr(A|B) = Pr(A)] .
Following are also equivalent to the above:
Pr(A, B) = Pr(A)Pr(B)
Pr(B|A) = Pr(B).
When more than two events are involved in a conditional probability, then the following
holds:
Pr(A, B|C) = Pr(A|B, C)Pr(B|C).
Conditional probabilities are particularly important for compound experiments.
Examples
Let us say we have two urns. The first one contains 2 red balls and 1 black balls, and the second one contains 1 red ball and 1 black ball. If we pick one of the urns randomly and then choose one of the balls randomly, what is the probability that it will be red? What is the sample space here? It is S = {1r, 1b, 2r, 2b}. Here, 1r, for example, stands for observing a red ball from urn 1.
What is the probability distribution among the outcomes? It is not straightforward to find as before anymore. In fact, this is a compound experiment comprising two experiments: 1) choosing the urn with sample space S1 = {1,2} and 2) choosing a ball within an urn with sample space S2 = {r, b}. Here, S is a compound sample space constructed from the CartesianproductofS1 andS2 asS=S1×S2 ={1,2}×{r,b}={1r,1b,2r,2b}.
Let us consider X1 be the random variable for the outcome of choosing an urn (over S1)
5
and X2 be the random variable for the outcome of choosing a ball (over S2). Then
Pr(1r)=Pr(X1 =1,X2 =r) = Pr(X1 =1)Pr(X2 =r|X1 =1) = Pr(1)Pr(r|1)
= 1/2 × 2/3 = 1/3.
In the same way, Pr(1b) = 1/2 × 1/3 = 1/6,Pr(2r) = 1/2 × 1/2 = 1/4 and Pr(2b) = 1/2 × 1/2 = 1/4.
Now, how do we specify the event in question (what is the probability that it will be red)? It is A = {1r, 2r}. So, Pr(A) = Pr(1r) + Pr(2r) = 7/12.
It is often more convenient to use trees for conditional probabilities:
P(urn1)=
1/2 1/2
urn 1
P(red | urn1)= 1/3 1/2 2/3
P(urn1, red)=1/3 1/6 1/4
urn 2
1/2
1/4
red
black
red
black
Bayes Theorem
Let us say we know the probability of an event B given another event A. Now we want to know the probability of A given B. Bayes theorem helps to find that. It is simply written as this:
Pr(A|B) = Pr(B|A)Pr(A). (4) Pr(B)
It can be easily derived from the definition of conditional probability.
Now, let us say, there are n mutually disjoint events Aj , where j = 1, 2, . . . , n and their
union is the sample space: ∪nj=1Aj = S. Then probability of any event B ⊂ S can be written in the following way:
Pr(B) = Pr(B ∩ S); as B is completely contained in S = Pr(B ∩ (∪j Aj )); as given above
6
= Pr(∪j(B∩Aj))
= Pr(B∩Aj); asallB∩Aj aredisjoint
j
= Pr(B|Aj)Pr(Aj). definition of conditional probability. j
The above result is sometimes called the law of total probability. Replacing P(B) in 4 with the above, we get a more general Bayes theorem:
Pr(B|Ai)Pr(Ai) Pr(Ai|B) = j Pr(B|Aj)Pr(Aj).
Let us look at a classic example of Bayes theorem. Let us say that for a drug test, it returns positive result for a drug user 99% of the time and produces a negative result for a non-user 95% of the time. Suppose that 1% of the population uses drug. Then what is the probability that an individual is a drug user given that she tests positive?
Here, the sample space is constituted of {user+,user−,nonuser+,nonuser−}. Here, user−, for example, stands for the event that an individual is a drug user and he tests nega- tive. We actually do not know the probability distribution here. Instead, we only know some of the conditional probabilities: Pr(+|user) = 0.99,Pr(−|nonuser) = 0.95,Pr(user) = 0.01. We want to find Pr(user|+). Using Bayes theorem, we get
Pr(+|user)Pr(user) Pr(+|user)Pr(user) + Pr(+|nonuser)Pr(nonuser) 0.99 × 0.01
=
=
=
Pr(user|+) =
0.99 × 0.01 + (1 − Pr(−|nonuser)) × (1 − Pr(user)) 0.0099
0.0099+(1−0.95)×(1−0.01) 0.0099
0.0099 + 0.05 × 0.99 0.0099
=
≈ 0.167.
0.0099 + 0.0495
this is somewhat surprising and interesting. Even though the stats on the test sounded good, it is still rather unlikely that the person is a drug user even when she tests positive. Bayes theorem helped us realize that.
7
Expectations
Random variables can have values that are numbers, like an urn number, or they may have non-numeric values, like the color of a ball. If a random variable has exclusively numeric outcomes, then we can talk about its expected value, or expectation. The expected value of a numeric random variable is a weighted average of its possible numeric outcomes, where the weights are the probabilities of the outcome occurring.
When all the outcomes are equally likely, the expectation is the same as the average of possible outcomes. For a discrete random variable (sample space is a finite set), its expectation is defined as
E[X] =. xPr(X=x). x∈S
As any function of a random variable is also a random variable, it is possible to calculate the expectation of a function of a random variable, too. It can be calculated in the following way:
E[g(X)] =. g(x)Pr(X=x). x∈S
There are certain properties of expectations:
E[X +c] = E[X +Y] = E[aX] =
E[X]+c; where c is not a r.v. of the model E[X]+E[Y]
aE[X]; where a is not a r.v. of the model.
In a certain lottery, it is 0.01% likely to win and the prize is 1000 dollars. The ticket price is 10 dollars. The expected monetary gain from the lottery is
E[X] = (1000 − 10) × 0.0001 + (−10) × 0.9999 = 0.099 − 9.999
= −9.9.
It is almost the same as giving ten dollars away.
8
Conditional Expectations
A conditional expectation of a random variable is the expected value of the variable given that an event is already known to have happened. For discrete variables:
E[X|Y=y] =. xPr(X=x|Y=y). x∈S
Examples
Let us say, in the two urns example, we get 10 dollars for observing a red ball and we get 5 dollars for observing a black ball. What is the expected gain of this experiment? Here, the sample space is S = {1r, 1b, 2r, 2b}.
E[gain(X)] = gain(x)Pr(X = x) x∈S
= gain(1r)Pr(X = 1r) + gain(1b)Pr(X = 1b) +gain(2r)Pr(X = 2r) + gain(2b)Pr(X = 2b)
= 10×1+5×1+10×1+5×1 3644
≈ 7.9167.
Now, let us consider that we can decide which urn to choose. What is the expected gain if we choose urn 1? Let us define another random variable Y for the urn chosen.
E[gain(X)|Y = 1] = =
= =
gain(x)Pr(X = x|Y = 1)
x∈S
gain(1r)Pr(X = 1r|Y = 1) + gain(1b)Pr(X = 1b|Y = 1)
+gain(2r)Pr(X = 2r|Y = 1) + gain(2b)Pr(X = 2b|Y = 1) 10×2/3+5×1/3+0+0
25/3.
that the expected gain of choosing urn 2 is 10 × 1 + 5 × 1 = 15/2. 22
Similarly, we can calculate
So, it tells that choosing the first urn is more profitable.
Once again, it is often easier to use trees to find such values. The following tree shows the expected gain at each decision point:
9
2 ⇥ à + ⇥ 5 3 3
urn1
⇥ à + ⇥ 5 urn2 2 2
⇥ 25 + ⇥ 5 2322
⇡ àà96à
1/2 1/2
= 25 32
= 5
2/3 1/3 1/2 1/2
10 5 10 5
red
black
red
black
Law of Total Expectation
Following is a very useful proposition often known as the law of total expectation: E[X] = E[X|Y = y]Pr(Y = y).
y
It is analogous to the law of total probability that we derived for the Bayes theorem. The derivation of it as follows:
xy
= x Pr(X = x|Y = y)Pr(Y = y); definition of conditional probability xy
= xPr(X=x|Y =y) Pr(Y =y)
yx
= E[X|Y = y]Pr(Y = y). y
xPr(X=x) x
E[X] =
= x Pr{[X = x] ∩ [Y = y]}; see the derivation of the law of total probability
10