STAT4207/5207 Homework assignment 1
Due by February 10, 2021. Make sure you justify your answer. Write down the complete
derivations for full credits. Note that questions in which you are asked to classify the states, you
must specify whether they are transient, null recurrent, positive recurrent and you must specify their
period. These concepts will be discussed in lecture 3-5.
Exercise 1
Suppose that whether it rains or not depends on previous weather conditions through the last three
days. Suppose that if it has rained for the past three days, then it will rain today with probability
0.7; if it did not rain for any of the past three days, then it will rain with probability 0.3; and in any
other case the weather today will, with probability 0.6, be the same as the weather yesterday.
a) Show how this system may be analyzed by defining an appropriate Markov chain. How many
states are needed?
b) Determine the one step transition probability matrix P for this Markov chain.
Exercise 2
Exercise 14 of chapter 4 of your textbook (11th or 12th edition).
Exercise 3
Suppose P ∈ Rk×k and Q ∈ Rk×k are stochastic matrices. Show that P · Q is also a stochastic
matrix.
Exercise 4
Suppose you have two urns with a total of 5 balls. At each step, one of the five balls is chosen at
random and switched from its urn to the other urn. Let Xn be the number of balls in the first urn
after n switches.
a) Is {Xn : n = 1, 2, . . .} a Markov Chain? Explain.
b) Define the state space and provide the one step transition matrix.
c) Draw the corresponding transition graph.
d) Classify the states. Are there any periodic states?
e) What is the probability that, given I have 3 balls in the first urn at the 10th turn, I will have 2
balls at the 12th turn?
Exercise 5
Consider the pessimistic covid-19 model from Lecture 3 with state space S = {0, 1, 2, 3} and transi-
tion matrix
P =

0.6 0.4 0.0 0
0.3 0.5 0.2 0
0 0.4 0.5 0.1
0 0 0 1

1
a) Draw the transition graph.
b) Classify the states. Are there any periodic states?
c) Does this chain have a steady state distribution? Explain why or why not. If you answered yes,
derive the steady state distribution.
d) Suppose P(X0 = 0) = 0.7,P(X0 = 1) = 0.1,P(X0 = 2) = 0.05. Find E[X2].
Exercise 6
Consider a Markov Chain with state space ={0, 1, 2, 3}
P =

0.3 0.7 0 0
0.2 0.5 0.3 0
0 0.3 0.4 0.3
0 0 0.6 0.4

a) Draw the transition graph.
b) Does this chain have a steady state distribution? Explain why or why not. If you answered yes,
derive the steady state distribution.
c) What is the expected time to visit state 1 for the first time, given the chain starts in state 0?
d) Find limn→∞ P(Xn+3 = 1, Xn+5 = 3|X0 = 2). Do not forget to justify your answer.
e) Let A = {0, 3}. What is the probability that the chain reaches state 2 before it reaches any state
in A, given the chain starts in state 1?
2