Transcript ST3236: Stochastic Process Tutorial

ST3236: Stochastic Process
Tutorial 3
TA: Mar Choong Hock
Email: [email protected]
Exercises: 4
Question 1
A markov chain X0,X1,… on state 0, 1, 2 has the transition
probability matrix
and initial distributions, p0 = P(X0 = 0) = 0.3,
p1 = P(X0 = 1) = 0.4 and p2 = P(X0 = 2) = 0.3.
Determine P(X0 = 0, X1 = 1, X2 = 2) and draw the statediagram with transition probability.
Question 1
P(X0 = 0,X1 = 1,X2 = 2)
= P(X0 = 0)P(X1 = 1 | X0 = 0)P(X2 = 2 | X0 = 0,X1 = 1)
= P(X0 = 0)P(X1 = 1 | X0 = 0)P(X2 = 2 | X1 = 1)
= p0 x p01 x p12
= 0.3 x 0.2 x 0
= 0.
Note: pij = P(Xn=j|Xn-1=i)
p00=0.1
0
p10=0.9
p02=0.7
p20=0.1
p01=0.2
1
2
p21=0.8
p11=0.1
p22=0.1
Question 2
A markov chain X0,X1,… on state 0, 1, 2 has the transition
probability matrix
Determine the conditional probabilities
P(X2 = 1,X3 = 1|X1 = 0) and P(X1 = 1,X2 = 1|X0 = 0).
Question 2
P(X2 = 1, X3 = 1 | X1 = 0)
= P(X2 = 1 | X1 = 1)P(X3 = 1 | X1 = 0, X2 = 1)
= P(X2 = 1 | X1 = 0)P(X3 = 1 | X2 = 1)
= p01 x p11
= 0.2 x 0.6
= 0.12
Similarly (or by stationarity),
P(X1 = 1, X2 = 1 | X0 = 0) = 0.12
In general,
P(Xn+1 = 1, Xn+2 = 1 | Xn = 0) = 0.12 for any n.
That is, it doesn’t matter when you start.
Question 3
A markov chain X0,X1,… on state 0, 1, 2 has the transition
probability matrix
If we know that the process starts in state X0 = 1, determine
probability P(X0 = 1,X1 = 0,X2 = 2).
Question 3
P(X0 = 1,X1 = 0,X2 = 2)
= P(X0 = 1)P(X1 = 0| X0 = 1)P(X2 = 2 | X0 = 1,X1 = 0)
= P(X0 = 1)P(X1 = 0| X0 = 1)P(X2 = 2 | X1 = 0)
= p1 x p10 x p02
= 1 x 0.3 x 0.1
= 0.03
Question 4
A markov chain X0,X1,… on state 0, 1, 2 has the transition
probability matrix
Question 4a
Compute the two-step transition matrix P(2).
Note: Observe that the rows must always sum to one for
all transition matrices.
Question 4b
What is P(X3 = 1|X1 = 0)?
P(X3 = 1|X1 = 0) = 0.13
In general,
P(Xn+2 = 1 | Xn = 0) = 0.13 for any n.
Question 4c
What is P(X3 = 1|X0 = 0)?
Note that:
Thus,
P(X3 = 1|X0 = 0) = 0.16
In general,
P(Xn+3 = 1 | Xn = 0) = 0.16 for any n.
Question 5
A markov chain X0,X1,… on state 0, 1, 2 has the transition
probability matrix
It is known that the process starts in state X0 = 1, determine
probability P(X2 = 2).
Question 5
Note that:
P(X2 = 2) = P(X0 = 0) x P(X2 = 2 | X0 = 0)
+P(X0 = 1) x P(X2 = 2 | X0 = 1)
+P(X0 = 2) x P(X2 = 2 | X0 = 2)
= p0p02 + p1p12 + p2p22
= 1 x p12 = 0.35
Question 6
• Consider a sequence of items from a production process,
with each item being graded as good or defective.
• Suppose that a good item is followed by another good
item with probability  and by a defective item with
probability 1-.
• Similarly, a defective item is followed by another defective
item with probability  and by a good item with probability
1-.
• Specify the transition probability matrix.
• If the first item is good, what is the probability that the first
defective item to appear is the fifth item?
Question 6
Let Xn be the grade of the nth product.
P(Xn+1 = g | Xn = g) = , P(Xn+1 = d | Xn = g) = 1 -
P(Xn+1 = d | Xn = d) = , P(Xn+1 = g | Xn = d) = 1 - 
Thus, the transition probability matrix is
Question 6
The probability is,
P(X5 = d,X4 = g,X3 = g,X2 = g | X1 = g)
= P(X2 = g | X1 = g) x P(X3 = g | X2 = g)
x P(X4 = g | X3 = g) x P(X5 = d | X4 = g) (why?)
= pgg x pgg x pgg x pgd
= 3(1 -)
pgd=1-
g
d
pdd= 
pgg=
pdg=1-
Question 7
The random variables 1, 2, ... are independent and with
common probability mass function
Set X0 = 0 and let Xn = max {1, 2, ... }.
• Determine the transition probability matrix for the MC
{Xn}.
• Draw the state-diagram associated with transition
probability
Question 7
Observe:
X0 = 0,
X1 = max {X0, 1},
X2 = max {X1, 2},
…
Xn= max {Xn-1, n}
Hence, Xn recursively compares the previous
maximum and the current input to obtain the new
maximum.
Question 7
The state space is S = {0, 1, 2, 3}
P(Xn+1 = 0 | Xn = 0) = P(n+1 = 0)= 0.1
P(Xn+1 = 1 | Xn = 0) = P(n+1 = 1)= 0.3
P(Xn+1 = 2 | Xn = 0) = 0.2
P(Xn+1 = 3 | Xn = 0) = 0.4
P(Xn+1 = 1 | Xn = 1) = P(n+1 = 0) + P(n+1 = 1)
= 0.1 + 0.3 = 0.4
P(Xn+1 = 2 | Xn = 1) = 0.2
…
P(Xn+1 = 2 | Xn = 2) = 0.1 + 0.3 + 0.2 = 0.6
…
P(Xn+1 = 3 | Xn = 3) = 0.1 + 0.3 + 0.2 + 0.4 = 1
…
P(Xn+1 = j | Xn = i) = 0 if j < i. (Cannot Happen!)
Question 7
The transition probability matrix is
p03=0.4
p02=0.2
p23=0.4
p01=0.3
0
p00=0.1
1
p11=0.4
2
p12=0.2
3
p22=0.6
p13=0.4
Note: pij = P(Xn=j|Xn-1=i)
p33=1