Chapter 4 Discrete time Markov Chain

• • •

Learning objectives :

Introduce discrete time Markov Chain Model manufacturing systems using Markov Chain Able to evaluate the steady-state performances

Textbook :

C. Cassandras and S. Lafortune, Introduction to Discrete Event Systems, Springer, 2007 1

Plan

• • • Basic definitions of discrete time Markov Chains Classification of Discrete Time Markov Chains Analysis of Discrete Time Markov Chains 2

Basic definitions of discrete time Markov chains

3

Discrete Time Markov Chain (DTMC) Definition

: a stochastic process with discrete state space and discrete time {X n , n > 0} is a discrete time Markov Chain (DTMC) iff P[X n+1 = j  X n = i n , ..., X 0 = i 0 ] = P[X n+1 = j  X n = i n ] = p ij (n) In a DTMC, the past history impacts on the future evolution of the system via the current state of the system p ij (n) is called

transition probability

from state i to state j at time n.

4

Discrete Time Markov Chain (DTMC) Discrete time Memoryless Discrete events Stochastic process Continuous event Continuous time

A DTMC is a discrete time and memoriless discrete event stochastic process.

5

Example: a mouse in a maze (

老鼠在迷宫 ) 1 2 start exit 5 3 0 4 Which stochastic process can be used to represent the position of the mouse at time t?

Under which assumptions, the system can be represented by a discrete time Markov chain?

6

Example: a mouse in a maze

1/2 1 exit 5 1/2 1 1/4 1/4 1/4 3 0 1 1/2 1/4 1 2 4 1/2 start • • Let {X n } n=0, 1, 2, ...

the position of the mouse after n rooms visited Assume that the mouse does not have any memory of rooms visited previously and that she chooses any corridor equi probably.

7

Homogenuous DTMC

• A DTMC is said homogenuous iff its transitions probabilities do not depend on the time n, i.e.

P[X n+1 = j  X n = i] = P[X 1 = j  X 0 = i] = p ij • A homogenuous DTMC is then defined by its transition matrix P =[p

ij

] i,j  E 8

What is the transition matrix of the process?

1/2 1 exit 5 1/2 1 1/4 1/4 1/4 3 0 1 1/2 1/4 1 2 4 1/2 start • • Let {X n } n=0, 1, 2, ...

the position of the mouse after n rooms visited Assume that the mouse does not have any memory of rooms visited previously and that she chooses any corridor equi probably.

9

Stochastic Matrix

A square matrix is said

stochastic

iff • all entries are non negative • each line sums to 1 • • •

Properties

: A transition matrix is a stochastic matrix If P is stochastic, then P n is stochastic The eigenvalues of P are all smaller than 1, i.e. | l | ≤1 10

Assumptions

• In the remaining of the chapter, we limit ourselves to Markov chain • • of discrete time defined on a finite state space E • homogeneous in time.

• Note that most results extend to countable state space.

11

Graphic representation of a DTMC

12

Classification of Discrete Time Markov Chains

13

Classification of states

• Let f jj be the probability of returning to state j after leaving j.

• • • A state j is said

transient

if f jj < 1 A state j is said

recurrent

if f jj = 1 A state j is said

absorbing

if p jj = 1.

• Let T jj be the average reccurn time, i.e. time of returning to j • • A recurrent state j is

positive recurrent

if E[T jj ] is finite.

A recurrent state j is

null recurrent

if E[T jj ] =  .

14

Classify the states of the example

1 exit 5 1/2 1 1/4 1/4 1/4 3 0 1 1/2 1/2 1/4 1 2 1/2 4 start 15

Irreducible Markov chain

• A DTMC is said

irreducible

iff a state j can be reached in a finite number of steps from any other state i.

• An irreducible DTMC is a strongly connected graph.

16

Irreducble Markov chain

17

Periodic Markov chain

• • A state j is said periodic if it is visited only in a number of steps which is multiple of an integer d > 1, called period.

A state j is said

aperiodic

otherwise •

A state with a self-loop transition

(i.e. p ii > 0) is always aperiodic.

• All states of an irreducible Markov chain have the same period.

18

Partitionning a DTMC into irreducible sub-chains

• A DTMC can be partitionned into strongly connected components, each corresponding to an irreducible sub-chain.

19

Classification of irreducible sub-chains

• • A sub-chain is said absorbing if there is no arc going out of it.

Otherwise, the sub-chain is transient.

transcient sub-chain absorbing sub-chain absorbing sub-chain 20

Canonic form of transition matrix

• • • Q : transitions of transient sub-chains Pi : transititions between states of aborbing sub-chain i Ri: Transitions toward absorbing sub-chain i 21

Formal definitions

• A state j is said

reachable

from a state i if there is a path from i to j in the state transition diagram.

• A subset S of states is said

closed

if there is no transition leaving S.

• A closed set S is said

irreducible

if all states in S are mutually reachable.

• A Markov chain is said irreducible if its state space is irreducible.

22

Theorems Th1.

If a Markov chain has a finite state space, then at least one state is recurrent.

Th2.

If i is a recurrent state and j is reachable from i, then state j is recurrent.

Th3.

If S is a finite closed irreducible set of states, then every state in S is recurrent.

Th4.

If i is a positive recurrent state and j is reachable from i, then state j is positive recurrent.

Th5.

If S is a closed irreducible set of states, then every state in S is positive recurrent or every state in S is null recurrent or every state in S is transient.

Th6.

If S is a finite closed irreducible set of states, then every state in S is positive recurrent.

23

Analysis of DTMC

24

Sojourn time in a state

• Let Ti be the temps spent in state i before jumping to other states.



i



n

 

p ii n

 1  1 

p ii

 • Ti is a random variable of

geometric distribution

.

25

Properties of geometric distribution

• Let X be a random variable of geometric distribution with parameter p, i.e. P{X = n} = (1-p) n-1 p.

• • • • • E[X] = 1/p Var(X) = 1/p 2 s X = 1/p Coefficient of variation = s X / E[X] = 1 Memoryless (only discrete distribution of this property):  

n

  

n



m

   1 

p

 1  

n m

1

p



n

 1

p



p



m

 1

p

 

m

 26

m-step transition probabilities

• The probability of going from i to j in m steps is p ij (m) = P{X n+m = j|X n =i} = P{X m = j|X 0 =i}.

• Let P (m) = [p ij (m) ] be the m-step transition matrix • • Properties (to prove): P (m) = P m Chapman-Kolmogorov equation: P (l+m) = P (l) P (m) or

p ij

   

kj

27

Example

1 exit 5 1/2 1 1/4 1/4 1/4 3 0 1 1/2 1/2 1/4 1 2 1/2 4 start • What is the probability that the mouse is still in room 2 at time 4? (p 22 (4) ) 28

Probability of going from i to j in exactly n steps

• f ij (n) : probability of going from i to j in exactly n steps (without passing j before) • f ij : probability of going from i to j in a finite number of steps

f ij f ij

 

n

   1

p ij f ij



k

 

j p f ik kj

• Similar approach can be used to determine the average time T ij it takes for going from i to j 29

Probability distribution of states

• • • • p i (n) : probability of being in state i at time n p i (n) = P{X n = i} p (n) = ( p 1 (n), p 2 (n), ...) : vector of

probability distribution

over the state space at time n The probability distribution p (n) depends on ─ the transition matrix P ─ the initial distribution p (0) Remark: if the system is at state i for certainty, then and p j (n) = 0, for j ≠i p i (0) = 1 • What is the relation between p (n), p (0), and P? 30

Transient state equations

• By conditioning on the state at time n,

Property

: Let P be the transition matrix of a markov chain and p (0) the initial distribution, then over the state space at time n p (n+1) = p (n)P p (n)= p (0)P n 31

Steady-state distribution

• • • •

Key questions

: Is the distribution p (n) converges when n goes to infinity?

If the distribution converges, does its limit p depend on the initial distribution p (0)?

= ( p 1 , p 2 , ...) If a state is recurrent, what is the percentage of time spent in this state and what is the number of transitions between two successive visits to the state?

If a state is absorbing, what is the probability of ending at this state? What is the average time to this state?

32

Steady state distribution

• •

Theorem

: For a irreducible and aperiodic DTMC with positive recurrent states, the distribution p (n) converges to a limit vector p which is independent of p (0) and is the unique solution of the system:

P

   p

i

 1    p 

j

 p

i

 p

i

 1

p ij

,

E

Normalization equation balance equation equilibrium equation p i are also called

stationary probabilities (also called steady state or equilibrium distribution)

.

For an irreducible and periodic DTMC, p i of time spent in state i are the percentage 33

Flow balance equation

• Equation p

j

  p

i p ij

can be interpretated as balance equation of probability flow.

• A probability flow p i p ij is associated to each transition (i, j).

•  p

i p ij

is the sum of probability flow into node j • p

j

or  p

j p ji

is the sum of flow out of node j • The flow balance equation : Outgoing flow = Incoming flow 34

A manufaturing system

• • • • Consider a machine which can be either UP or DOWN.

The state of the machine is checked every day.

The average time to failure of an UP machine is 10 days.

The average time for repair of a DOWN machine is 1.5 days.

• • • • • • Determine the conditions for the state of the machine {X n } at the begining of each day to be a Markov chain.

Draw the Markov chain model.

Find the transient distribution by starting from state UP and DOWN.

Check whether the Markov chain is recurrent and aperiodic.

Determine the steady state distribution.

Determine the availability of the machine.

35

A telephone call process

• • • • • Discrete time model with time slots indexed by k = 0, 1, 2, ...

At most one telephone call can occur in a single time slot, and there is a probability a that a call occurs in any slot If the phone is busy, the call is lost; otherwise, the call is processed.

There is a probability b that a call in process completes in any time slot If both a call arrival and a call completion occur in the same time slot, the new call will be processed.

• •

Issues to solve:

Markov chain model Loss probability 36