MAST30001 Stochastic Modelling
Tutorial Sheet 8
1. A CTMC (Xt)t≥0 with state space S = {1, 2, 3, 4} has non-zero transition rates
q1,2 = 4, q2,1 = 1 = q2,4 and q2,3 = 3. Suppose that P(X0 = 1) = 1 (i.e. the chain
starts in state 1), and let T1 = inf{t > 0 : Xt 6= X0} be the first jump time of
(Xt)t≥0, and T2 = inf{t > T1 : Xt 6= XT1} denote the time of the second jump.
(a) Draw the transition diagram for the CTMC (Xt)t≥0
(b) Describe the communicating classes of (Xt)t≥0
(c) Find h1,3, the probability of ever reaching state 3.
(d) What is the distribution of T1?
(e) What is the distribution of XT1?
(f) Find E[T2].
(g) What is the distribution of XT2?
(h) Find m1,{3,4}, which is the expected time until (Xt)t≥0 reaches state 3 or 4.
(i) Find the limiting proportion of time spent in each state.
2. Let λ, µ > 0, and consider a CTMC with state space S = Z+ whose non-zero
transition rates are qi,i+1 = λ and qi+1,i = (i+ 1)µ for each i ∈ Z+.
(a) Explain intuitively why this CTMC is positive recurrent.
(b) Find the stationary distribution.
(c) Find the limiting distribution, starting from initial distribution a.
(d) Find the limiting proportion of time spent in each state.
3. (CTMCs as limits of DTMCs) Let P be a stochastic matrix with i, j-th entry pi,j,
and such that pi,i = 0 for all i. For (λi)i∈S and for each integer m > supi∈S λi, define
a DTMC (Y (m)n )n∈Z+ by
P(Y (m)n+1 = i|Y
(m)
n = i) =
(
1−
λi
m
)
,
and for i 6= j
P(Y (m)n+1 = j|Y
(m)
n = i) =
λi
m
pij.
Define a continuous time process (not a CTMC though) by
X
(m)
t = Y
(m)
bmtc,
where bac is the greatest integer not bigger than a.
(a) What does a typical trajectory of X (m) look like? At what times does it jump?
(b) Given X
(m)
0 = i, what is the distribution of the random time
T (m)(i) = min{t ≥ 0 : X (m)t 6= i}
(c) As m→∞, to what distribution does that of the previous item converge?
(d) It turns our that X (m) converges (in a certain sense) as m→∞ to a continuous
time Markov chain. What are its transition rates?