Transcript Markov Chain - Universitas Gadjah Mada

Markov Chain
Nur Aini Masruroh
Discrete Time Markov Chains
Consider a stochastic process {Xn, n=0, 1, 2, …} that takes
on a finite or countable number of possible values
– assume set of possible values are non negative integers
{0, 1, 2, …}
• Let Xn= i denote the process in state i at time n
• {Xn, n=0, 1, 2, …} describes the evolution of process over
time
• Define Pij as the probability that the process will next be
in state j given that it is currently in state i Pij = P{Xn+1 =
j | Xn= i}
• Such a stochastic process is known as a Discrete Time
Markov chain (DTMC)
Discrete Time Markov Chains
DTMC can be used to model a lot of real life
stochastic phenomena
– Example: Xn can be the inventory on-hand of a
warehouse at the nth period
– Example: Xn can be the amount of money a taxi driver
gets for his nth trip
– Example: Xn can be the status of a machine on the nth
day of operation
– Example: Xn can be the weather (rain/shine) on the nth
day
DTMC properties
Markov Property:
 The conditional distribution of any future state Xn+1 given the past
state X0, X1, …, Xn-1 and the present state
 Xn, is independent of the past states and depends only on the
present state
P{Xn+1 = j | Xn=i, Xn-1=in-1, …, X1=i1, X0= i0}=P{Xn+1 = j | Xn=i}=
Pij
– Pij represents the probability that the process will, when in state i,
next make a transition into state j
 Since probabilities are non negative and since the process must
make a transition
into some state, we have that

Pij≥0, i,j ≥ 0;  pij  1
j 0
i = 0, 1, …
Short term analysis
 The N2 transition probabilities can be
represented by matrix P with element pij;
 0 ≤ pij ≤ 1
 Sum of the row should be 1
 p11 p12  p1N 
p

p

p
22
2N 
P  { pij }   21




p
p

p
N2
NN 
 N1
Defining a DTMC
 Specify the state
 State, s
• Variable to describe the present situation of the system,
i.e. pressure, temperature, etc
 Demonstrate the Markov property
 Find the probability transition
Example 1: rain
Suppose the chance of rain tomorrow depends on previous
weather conditions only through whether or not it rains
today, and not the past weather conditions. Suppose
also that if it rains today, then it will rain tomorrow with
probability α; and if it does not rain today, then it will
rain tomorrow with probability β. Let 0 be the state when
it rains and 1 when it does not rain. Model this as a
DTMC!
Example 2: stock movement
Consider the following model for the value of a stock:
 At the end of a given day, the price is recorded
 If the stock has gone up, the probability that it will go up
tomorrow is 0.7
 If the stock has gone down, the probability that it will go
up tomorrow is only 0.5 (assume stock staying the same
as a decrease)
 Model the price movement (up/down) as a DTMC
Example 3: Mood
 On any given day Gary is either cheerful (C), so-so (S),
or glum (G).
 If he is cheerful today, then he will be C, S or G
tomorrow with respective probabilities 0.5, 0.4, 0.1.
 If he is so-so today, then he will be C, S, or G tomorrow
with probabilities 0.3, 0.4, 0.3.
 If he is glum today, then he will be C, S, or G tomorrow
with probabilities 0.2, 0.3, 0.5.
 Model this as a DTMC
Example 4: Gambler’s ruin
 My initial fortune is $1 and my opponent’s
fortune is $2. I win a play with probability p, in
which case I receive $1 and my opponent loses
$1. We play until one of us have fortune 0.
 Model this as a DTMC
Example 5: beer branding
 A leading brewery company in Singapore (label T) has asked its IE
team to analyze its market position. It is particularly concerned about
its major competitor (label A).
 It is believed (and somewhat verified by consumer surveys) that
consumers chose their brand preference based on the brand of beer
they are currently consuming.
 From market survey data collected monthly,
 95% of the current consumers of label T will still prefer label T in the
next month, while 3% will switch to label A and the remaining to label C
(all other foreign brands)
 90% of consumers of label A will remain loyal to label A, while 8% will
shift preferences to label T
 80% of consumers of label C will prefer label C, while 10% will shift
preferences to label T.
 Model the brand loyalty/switching of consumers as a DTMC
State transition diagram
 The transition matrix of the Markov chain can be
represented by a state transition diagram (“bubble”
diagram)
 In the diagram:
 circles = nodes = states: one for each element of the state space
 arrows = archs = transitions: one for each non-zero probability
(labeled with probabilities)
 Diagram Rule: arrows out of a state have label sums
that add to 1.
 Try to draw state transition diagram for the Gambler’s
ruin and the beer branding examples!
Assignment 2
 Consider an experiment in which a rat is wandering inside a maze.
The maze has six rooms labeled F, 2, 3, 4, 5 and S. If a room has k
doors, the probability that the rat selects a particular door is 1/k.
However, if the rat reaches room F, which contains food, or room S,
which gives it an electrical shock, then it is kept there and the
experiment stops.
 Model the rat’s movement around the maze as a DTMC
 Draw the state transition diagram!
Markov process behavior
 Multistep transition probabilities
 Φij(n): probability that the process will occupy state j
at time n given that it occupied state i at time 0 (nstep transition probability from state i to state j)
Φij(n) = P{s(n) = j|s(0) = i}, 1 ≤ i, j ≤ N, n=0,1,2,…
 Probability in state j after n+1 transition:
P{s(n+1)=j|s(0)=i}
rewritten in terms of joint probability that state j is
occupied at time n+1 and state k is occupied at
time n:
N
P{s(n  1)  j | s(0)  i}   P{s(n  1)  j, s(n)  k | s(0)  i}
k 1
Multistep transition probabilities
From the definition of conditional probability:
N
P{s(n  1)  j | s(0)  i}   P{s(n)  k | s(0)  i}
k 1
 P{s(n  1)  j | s(n)  k , s(0)  i}
If n≥1, by Markov assumption that the future
trajectory only depends on the present state,
P{s(n  1)  j | s(n)  k , s(0)  i}
 P{s(n  1)  j | s(n)  k}  pkj
Multistep transition probabilities
Based on the Markov property, the conditional
probability can be rewritten:
N
 ij (n  1)    ik (n) pkj
n  0,1,2,...
k 1
0   ij (n)  1 1  i, j  N , n  0,1,2,...
N

j 1
ij
(n)  1 i  1,2,..., N , n  0,1,2,...
Multistep transition probabilities
 Matrix formulation
11 (n) 12 (n)  1N (n) 
 ( n )  ( n )   ( n ) 
22
2N

 (n)  { ij (n)}   21




 N 1 (n)  N 2 (n)   NN (n) 
Φ(n+1) = Φ(n)P
Φ(0) = I  identity matrix
Φ(1) = Φ(0)P = IP = P
Φ(2) = Φ(1) P = P2
Φ(n) = Pn
n=0,1,2,…
Marketing example
In a hypothetical market there are only 2 brands, A and B.
A typical consumer in this market buys brand A with
probability of 0.8 if his last purchase was brand A and
with probability of 0.3 if his last purchase was brand B.
How does the probability of the consumer’s purchasing
each brand depend on the number of purchases he has
made and on the brand he purchased initially?
State probabilities
Suppose the probability that state I is occupied at time n will be given
the symbol πi(n) and defined
πi(n) =P{s(n)=i} i=1,2,…,N, n=0,1,2,…
N
  ( n)  1
i 1
i
We have Φij(n) = P{s(n) = j|s(0) = i}
If we multiply both side by P{s(0)=i} and sum over I from 1 to
N, we obtain
N
 P{s(0)  i}
i 1
N
  (0)
i 1
i
i 1
i
ij
(n)   P{s(0)  i}P{s (n)  j | s(0)  i}
i 1
N
ij
N
  (0)
N
(n)   P{s(0)  i, s(n)  j}  P{s(n)  j}
i 1
N
ij
(n)   j (n) or  j (n)    i (0) ij (n)
i 1
State probabilities
 The state probabilities at time n can be determined by multiplying
the multistep transition probabilities of starting in each state and
summing all over state
π(n) = [π1(n), π2(n), …, πN(n)]
with this definition,
π(n) = π(0)Φ(n)
n=0,1,2,…
π(n) = π(0)Pn
π(n+1) = π(0) Pn+1 = π(0)PnP
π(n+1) = π(n) P
n=0,1,2,…
 The state probability vector at any time can be calculated by postmultiplying the state probability vector at the preceding time by P
 Try the marketing example!
Asymptotic behavior of state probabilities
 When the process is allowed to make a large
transition,
π(∞) = π(0) P∞
If we define limiting state probability as π, so
π = π(0)Φ
Where Φ is the limiting multistep transition
probability matrix, Φ = Φ(∞) = P∞
Long term behavior and analysis
• In designing physical systems, there are often “start-up”
effects that are different from what can be expected in
the “long-run”.
– A designer might be interested in the long-run
behavior, or operations
• The LLN holds for iid random variables.
• Question: Do similar limiting results hold for DTMC when n
is large?
– Limiting distributions
– Long term averages
Limiting probabilities
 Monodesmic process: Markov process that has a Φ with equal
rows
 Monodesmic process:
 Sufficient condition: able to make transition
 Necessary condition: exit only one subset of states that must be
occupied after infinitely many transitions
 Recall:
N
 j    i (0)ij
j  1,2,...,N
i 1
Because all rows of Φ are equal for a monodesmic process,
each element Φij is equal to a value Φj that depends only on the
column index j. Then
N
N
i 1
i 1
 j    i (0) j   j   i (0)   j
Direct solution for limiting state
probabilities
 If the state probability vector has attained its limiting value, π, it
must satisfy the equation
π = πP
It implies the N simultaneous equations:
N
 j    i pij
i 1
N

i 1
i
1
Try the marketing example!
j  1,2,..., N
Example 1: rain
Suppose the chance of rain tomorrow depends on previous
weather conditions only through whether or not it rains
today, and not the past weather conditions. Suppose
also that if it rains today, then it will rain tomorrow with
probability α; and if it does not rain today, then it will
rain tomorrow with probability β. Let 0 be the state when
it rains and 1 when it does not rain. Find the limiting
probabilities π0, π1 .
Example 2: Tank warfare
One theoretical model for tank warfare expresses the firing mechanism
as a two-state Markov process in which the states are 0: a hit, 1: a
miss. Thus Xn is 1 or 0 depending on whether the nth shot is a miss
or a hit. Suppose the probability of the tank’s hitting on a certain shot
after it had hit on the previous shot is ¾, and the probability of the
tank’s hitting on a certain shot after it had missed on the previous
shot is ½,
– find the probability that the 11th shot fired from the tank (the 10th shot
after the first) hits its target, given that the initial shot hit.
– Suppose that a tank commander on first encountering an enemy fires
his first round for ranging purposes, and suppose that it is fairly
unlikely that the first round hits. More specifically, suppose that the
probability of a hit on the initial shot is ¼, and that the probability of a
miss on the initial shot is ¾. What is the probability that the fourth
shot hits?