Assignment 1 (for Stat3021)
1. Assume Xn are independent and identically distributed with P (X1 = 1) = p,
P (X1 = 0) = r and P (X1 = −1) = q. where p, r, q > 0 and p + r + q = 1.
Let Sn =
∑n
i=1Xi, n = 1, 2, . . ..
(a) Prove that {S1, S2, . . .} is an irreducible Markov chain with state space S =
{0,±1,±2, ...} and write down its transition matrix.
(b) Is the chain aperiodic?
(c) Find expressions for:
i. P (S3 = 2).
ii. P (S4 = 1|S1 = 1).
iii. P (S10 = 1|S7 = 0).
iv. ESn and var(Sn).
2. Suppose a transition diagram is given as follows:
1 2
5
3
4 6 7
1/2
1/2
1/2
1/4
1/4
For this chain, there are two recurrent classes R1 = {6, 7} and R2 = {1, 2, 5}, and
one transient class R3 = {3, 4}.
(a) Find the period of state 3 and the probability that, starting at state 3, the
state 3 is eventually re-entered (i.e., f33).
(b) Assuming X0 = 3, find the probability that the chain gets absorbed in R1.
(c) Assuming X0 = 3, find the expected time (number of steps) until the chain
gets absorbed in R1 or R2. More specifically, let T be the absorption time,
i.e., the first time the chain visits a state in R1 or R2. We would like to find
E[T |X0 = 3].
Note: there are missing transition probabilities for this chain, but no impact for
your solution.
3. An office worker owns 3 umbrellas which she uses to go from home to work and vice
versa. If it is raining in the morning she takes an umbrella (if she has one) on her
trip to work and if it is raining at night she takes an umbrella (if she has one) on
her trip home. If it is not raining, she doesn’t take an umbrella. At the beginning
of each trip it is raining with probability p = 1− q, independently of previous trips.
Let Xn represent the number of umbrellas available at the beginning of the n-th
trip and assume that 0 < p < 1.
(a) Explain why {Xn}n≥1 is a Markov Chain with state space S = {0, 1, 2, 3}
having the transition probability matrix P :
P =

0 0 0 1
0 0 1− p p
0 1− p p 0
1− p p 0 0
 .
(b) Classify the states of the Markov chain in (a) under additional condition 0 ≤
p ≤ 1.
(c) Explain or calculate directly why P (X3 = 0 | X1 = 0) = 1− p.
(d) If no umbrellas are available at the beginning of the first trip, calculate the
average number of trips until there are again no umbrellas available.
(e) Explain why the limits of P (Xn = i) as n → ∞ exists, for i = 0, 1, 2, 3, and
calculate them.
(f) After a large number of trips, how many umbrellas are available on average at
the beginning of a trip?
(g) In a large number of trips, about what proportion of journeys does she get
wet?