COMP2610/COMP6261 – Information Theory
Tutorial 1: Probability and Bayesian Inference
Robert C. Williamson
Week 1, Semester 2, 2018
1. (From Bishop, 2006) Suppose that we have three coloured boxes r (red), b (blue), and g (green). Box r
contains 3 apples, 4 oranges, and 3 limes. Box b contains 1 apple, 1 orange, and 0 limes. Box g contains 3
apples, 3 oranges, and 4 limes. A box is chosen at random with probabilities p(r) = 0.2, p(b) = 0.2, p(g) =
0.6 and a piece of fruit is removed from the box (with probability of selecting any of the items in the box):
(a) What is the probability of selecting an apple?
(b) If we observe that the selected fruit is in fact an orange, what is the probability that it came from the
green box?
2. A scientist conducts 1000 trials of an experiment involves variablesX,Y , each with possible values {0, 1}.
He records the outcomes of the trials in the following table.
Counts X
0 1
Y
0 100 250
1 150 500
Compute each of the following, showing all your working.
(a) p(X = 1, Y = 1).
(b) p(X = 1).
(c) E[X].
(d) p(Y = 1|X = 1).
(e) p(Y = 1|X = 0).
(f) Let Z be a noisy version of the XOR of X and Y , with
p(Z = 1|X = x, Y = y) =
{
0.9 if (x, y) = (0, 1) or (x, y) = (1, 0)
0.1 if (x, y) = (0, 0) or (x, y) = (1, 1).
Compute p(X = 1, Y = 1|Z = 1).
3. (From Barber, 2011) Two balls are placed in a box as follows:
• a fair coin is tossed
• a white ball is placed in the box if a head occurs; otherwise, a red ball is placed in the box
• the coin is tossed again
• a red ball is placed in the box if a tail occurs; otherwise, a white ball is placed in the box.
Balls are drawn from the box three times in succession (always with replacing the drawn ball back in the
box). It is found that on all three occasions a red ball is drawn. What is the probability that both balls in the
box are red?
1
4. Several of the results we have seen in lectures generalise in the expected way when we condition on additional
random variables.
(a) For random variables X,Y , the conditional probability p(X|Y ) may be defined as
p(X|Y ) =
p(X,Y )
p(Y )
.
Using this definition, show that for random variablesX,Y, Z, the following conditional version of Bayes
rule holds:
p(X|Y, Z) =
p(Y |X,Z)p(X|Z)
p(Y |Z)
.
(b) The sum rule (or marginalisation) says that for random variables X1, . . . , Xn,
p(X1, . . . , Xi−1, Xi+1, . . . , Xn) =
∑
x
p(X1, . . . , Xi−1, Xi = x,Xi+1, . . . , Xn).
Using this fact, and the definition of conditional probability, show that
p(X1, . . . , Xi−1, Xi+1, . . . , Xn|Y ) =
∑
x
p(X1, . . . , Xi−1, Xi = x,Xi+1, . . . , Xn|Y )
where Y is another random variable.
5. (From Pearl, 1988) Three prisoners A, B and C are being tried for murder. Their verdicts will be read and
their sentences executed tomorrow. They know only that one of them will be declared guilty and will be
hanged while the other two will go free; the identity of the condemned prisoner is revealed to a reliable
prison guard, but not to the prisoners.
In the middle of the night prisoner A makes the following request. “Please give this letter to one of my
friends – to one who is to be released. You and I know that at least one of them will be released”. The guard
carries out this request. Later, prisonerA calls the guard and asks him to whom he gave the letter. The guard
tells him that he gave the letter to prisoner B.
What is the probability that prisoner A will be released?
2