CTMC

Transcript CTMC

Chapter 5 Continuous time Markov Chains

Learning objectives :

Introduce continuous time Markov Chain Model manufacturing systems using Markov Chain Able to evaluate the steady-state performances

Textbook :

C. Cassandras and S. Lafortune, Introduction to Discrete Event Systems, Springer, 2007 1

Plan

• • • • • Basic definitions of continuous time Markov Chains Characteristics of CTMC Performance analysis of CTMC Poisson process Approximation of general distributions by phase type distribution 2

Basic definitions of continuous time Markov Chains

Continuous Time Markov Chain (CTMC) Discrete time Discrete events Stochastic process Continuous event Continuous time Memoryless

A CTMC is a continuous time and memoriless discrete event stochastic process.

Continuous Time Markov Chain (CTMC) Definition

: a stochastic process with discrete state space and continuous time {X(t), t > 0} is a continuous time Markov Chain (CTMC) iff P[X(t+s)= j  X(u), 0≤u≤s] = P[X(t+s)= j  X(s)], " t, " s, " j

Memoryless

: In a CTMC, the past history impacts on the future evolution of the system via the current state of the system 5

Continuous Time Markov Chain (CTMC)

Poisson Arrivals Exponential service time

N(t)

: number of customers at time t Customer Arrivals Customer departures 6

Homogenuous CTMC Definition

: A CTMC {X(t), t > 0} is

homogeneous

iff P[X(t+s)= j  X(t) = i] = P[X(t+s)= j  X(t) = i] = p ij (s)

Homogeneous memoryless

: In reliability, we only say "a machine that does not fail at age t is

as good as new

Only homogeneous CTMC will be considered in this chapter

Characteristics of CTMC

X(t)

Behavior of a CTMC

• • Two major components: T i = sojourn time in state i (random variable) p ij = probability of moving to state j when leaving state i 9

Sojourn time in a state

• Let T i be the random variable corresponding to the time spent in state i • The memoryless property of the homogenuous CTMC implies 

i i

 



 ,

t x

• The exponential distribution is the only continuous probability distribution having this property.

In an CTMC, the sojourn time in any state is exponentially distributed.

Exponential distribution

• Let T be a continuous random variable with an

exponential distribution of parameter

l • • • • Distribution Function (figure) : F T (t) = P{T ≤ t}

     1 0,

 l

t t

 0  0 Probability density function : f T (t) = dF T (t)/dt

f T

  l 0,

 l

 0

 0 Mean : E[T] = 1/ l Standard deviation: s [T] = 1/ l • Coeficient of variation: Cv(T) = s [T]/ E[T] = 1 •

Parameter

l often corresponds to some

event rate

(failure rate, repair rate, production rate, ...) 11

Exponential distribution

• Memoryless :  

 l



 l 

 l

 

t t



 l

 

 • For a machine with exponentially distributed lifetime, we say that it is "

as good as new

" if it is not failed. • The remaining lifetime of an used but UP machine has the same distribution as a new machine. 12

Transition probability

Whe a CTMC leaves state i, it jumps to state j with • • probability p ij . This probability is: independent of time as the CTMC is homogeneous independent of sojourn time Ti as the process is markovian (memoryless) 13

1st characterization of a CTMC

An CTMC is fully characterized by the following • • parameters: { m i } i  E with m i as the parameter of the exponential distribution of sojourn time T i {p ij } i≠j , with pij as the transition probability from i to j when leaving state i 14

Classification of a CTMC

Each CTMC is associated an underlying DTMC by neglecting sojourn times.

A state i of a CTMC is said transient (resp. recurrent, absorbing) if it is transient (resp. recurrent, absorbing) in the underlying DTCM A CTMC is irreducible if its underlying DTMC is irreducible.

Remark: the concept of periodicity is not relevant.

2nd characterization of a CTMC

Each state activates several potential events leading to different transitions.

A CTMC travels from state i to state j in T ij time, an exponentially distributed random variable with parameter m ij . m i is called

transition rate

from i to j.

Equivalence of the two representation

• • Let T i p ij = MIN j {T ij } = P{T ij = T i } Result to prove: T i = EXP( Sm ij ), p ij is independent of T i Moment generating function M X (u) = E[exp(uX)] 17

Performance analysis of CTMC

Probability distribution

• State probability p i (t) = P{X(t) = i} • state probability vector, also called probability distribution p (t) = ( p 1 (t), p 2 (t), ...) 19

Transient analysis

By conditionning on X(t), With 20

It can be shown,

Transient analysis

Letting dt go to 0, 21

Infinitesimal generator

• Let • The matrix Q = [q ij ] is called infinitesimal generator of the CTMC • As a ressult, 22

Steady state distribution of a CTMC Thereom

: For an irreducible CTMC with postive recurrent states, the probability distribution converges to a vector of stationary probabilities ( p 1 , p 2 , ...) that is independent of the initial distribution p (0). Further it is the unique solution of the following equation system: normalization equation flow balance equation or equilibrium eq 23

Flow balance equation

• The balance equation equivalent to : S i≠j p j m ji = S i≠j p i m ij • Associate to each transition (i,j) a

probability flow

: p i m ij • • S i≠j S i≠j p j m ji p i m ij : total flow into state i : total flow out of state i • Interpretation : Total flow in = Total flow out 24

Flow balance equation of set of states

• Let E 1 be a subset of states • Flow balance equation : Total flow into E 1 = Total flow out of E 1 25

A manufaturing system

• • • • Consider a machine which can be either UP or DOWN.

The state of the machine is checked continuously.

The average time to failure of an UP machine is 10 days.

The average time for repair of a DOWN machine is 1.5 days.

• • • • • • Determine the conditions for the state of the machine {X(t)} to be a Markov chain.

Draw the Markov chain model.

Find the transient distribution by starting from state UP and DOWN.

Check whether the Markov chain is recurrent.

Determine the steady state distribution.

Determine the availability of the machine.

Poisson process

Poisson process

• • A Poisson process is a stochastic process N(t) such that N(0) = 1 N(t) increments by +1 after a time T random distributed according to an exponential distribution of parameter l .

An arrival process is said Poisson if the inter-arrival times are exponentially distributed.

Properties of Poisson process

A Poisson process is an irreducible CTMC N(t) has a Poisson distribution with parameter l t 29

Properties of Poisson process

A Poisson process is an irreducible CTMC P{N(t+dt) = k+1 | N(t) = k} = l dt + o(dt) Probability of 0 arrival in dt P{N(t+dt) = k | N(t) = k} = 1 l dt + o(dt) Probability of more than one arrival in dt P{N(t+dt) > k+1 | N(t) = k} = o(dt) 30

Properties of Poisson process

The superposition of n Poisson process of parameter l i Poisson process of parameter Sl i is a Assume that a Poisson process is split into n processes with probabilities p i . These n process are independent Poisson process with parameter l p i 31

Birth-Death process

Definition

• Consider a population of individuals • Let N(t) be the size of the population with N(t) = 0, 1, 2, ...

• When N(t) = n,

births rate

l n > 0 arrive at according to a Poisson pocess of

birth

• Deaths arrive also according to a Poisson process of

death rate

m n > 0.

Key issues

• Graphic representation of the Markov chain • Relation with the Poisson process (also called pure birth process) • • • Condition for existence of steady state distribution

  

 1 l l 0 m m 1

 1

  Sufficient condition (larger death rate than birth rate) m l

n n

 1 

n n

* Steady state distribution p n 34

Approximation of general distributions by phase type distribution

Phase-type distribution A probaiblity distribution that results from a system of one or more inter-related Poisson process

occurring in sequence, or phases.

The sequence

in which each of the phases occur

may itself be a stochastic process

Phase distribution

time until the absorption of a CTMC one absorbing state

. Each of the states of the Markov process represents one of the phases.

Phase-type distributions

can be used to approximate any positive valued distribution

Definition

• A CTMC with

+1 states, where

≥ 1, such that the states 1,...,

are transient states and state

+1 is an absorbing state. • An initial probability of starting in any of the

+1 phases given by the probability vector (

, α

+1 ).

The

continuous phase-type distribution

is the distribution of time from the above process's starting until absorption in the absorbing state.

This process can be written in the form of a transition rate matrix,



S 0 S

0 0   where

is an m×m matrix and

0 = -

S 1

with

represents an

×1 vector with every element being 1 37

Characterization

Time

until the absorbing state is phase-type distributed PH(

The distribution function of

is given by, F(

) = 1 a exp( Sx )

, and the density function, f(

) = a exp( Sx )

S 0

, for all

> 0. 38

Erlang distribution E k : k-stage Erlang distribution with parameter

X = sum of k independent random variable of exponential distribution with parameter

E[X] = k/

Var[X] = k/

2 C X =

X / E[X] = 1/k 1/2

m m ●●● m 39

Hyper-exponential or mixture of exponential distribution X =

1 X 1 +

2 X 2

• • where a 1 X i + a 2 ... + a n = EXP( m i )

... +

n X n

= 1,

E[X] =

1 /

1 + Var[X] =

1 /

1 2

2 /

2 ... +

n /

n +

2 /

2 2 ... +

n /

n 2

Coxian distribution

Coxian distribution can be used to approximate any distribution.

1-p 1 m1 p1 m2 p2 ●●● p n-1 1 m n 1-p 2 41

A manufaturing system

• • • • Consider a machine which can be either UP or DOWN.

The state of the machine is checked continuously.

The average time to failure of an UP machine is 10 days.

The average time for repair of a DOWN machine is 1.5 days.

• Assumed that UP time = E 2 and DOWN time = E 3 .

• Draw the Markov chain model.