MAST30001 Stochastic Modelling
Tutorial Sheet 4
1. Let p ∈ (0, 1) and suppose that a time homogenous Markov chain on Z = {. . . ,−2,−1, 0, 1, 2, . . .}
has transition probabilities given by
pi,i+2 = p, pi,i−1 = 1− p and pi,j = 0 for all i ∈ Z, j 6= i+ 2, i− 1.
(a) Using Stirling’s approximation, (for n ≥ 1)
1 ≤
n!en
nn
√
2πn
≤ 2,
determine for what values of p the chain is recurrent.
(b) Compute the probability of reaching state -1, starting in state 0.
2. Consider a Markov chain with state space S = {1, 2, . . . , 6} and transition matrix
given by
P =
1
2
1
2
0 0 0 0
1 0 0 0 0 0
0 3
4
0 1
4
0 0
0 0 1
2
0 1
2
0
0 0 0 0 0 1
0 0 0 0 2
3
1
3
(a) Find all stationary distributions for this chain
(b) Find the probability that we ever reach state 1, starting from state 3
(c) Find the expected time that we first reach the set of states A = {2, 5}, starting
from state 3.
(d) Starting from state 3, find the limiting proportion of time spent in state 1.
(e) Starting from state 3, find the expected limiting proportion of time spent in
state 1.
(f) Repeat the above two questions, starting from state 4 instead of state 3.
3. Assume that we have an infinite supply of a certain electrical component and that
the lifetimes (measured from the beginning of use) of the components are i.i.d.
N-valued random variables (so time is measured in discrete units) with
P(Ti = k) = qk > 0, k = 1, 2, . . . ,
and E[T1] <∞. At time n = 0, the first component begins use and when it fails (at time T1), it is immediately replaced by the second component, and when it fails (at time T1 +T2), it is replaced, and so on. Let Xn be the age of the component in use at time n and say Xn = 0 at times n where there is a failure. (a) Show that Xn is a Markov chain and write down its transition probabilities. (b) Explain why this chain is positive recurrent. (c) Find all equilibrium distributions for this Markov chain.