Question
Markov Chain and Invariant Distribution Question
Suppose you are given a deck of m cards and the cards are shuffles according to the process given as follows: at each time, t, the first card is taken off the top of the deck and inserted back at random into the deck. (In other words, the card is equally likely to land at any of the m positions possible in the deck, with position 1 the top of the deck and position m at the bottom.
a) Provide a Markov Chain description for the evolution of the deck of cards.
b) Establish that the Markov chain is irreducible and aperiodic.
c) Find the invariant distribution for the Markov Chain and explain why it is unique.
Explanation / Answer
Answer 1. A Markov chain, named after Andrey Markov, is a random process that undergoes transitions from one state to another on a state space 1.00 A Markov chain is a stochastic process with the Markov property. The term "Markov chain" refers to the sequence of random variables such a process moves through, with the Markov property defining serial dependence only between adjacent periods 2.00 It can thus be used for describing systems that follow a chain of linked events, where what happens next depends only on the current state of the system A Markov chain is a sequence of random variables X1, X2, X3, ... with the Markov property, namely that, given the present state, the future and past states are independent. "Random-to-top" is not an especially efficient way to randomize a deck of cards, but it is one of the easiest of all shuffling schemes to analyze mathematically. It works as follows. Assume that the deck consists of N cards, labelled 1, 2, 3, . . . , N , and assume that initially the cards are in order. At each time t = 1, 2, . . . , select a card at random, independently of all past selections, and move it to the top of the deck. Answer 2. Markov chain is irreducible and aperiodic. Proposition: Let (Xt , Yt ) be a Markov chain on X × X with transition probability kernel q. Then each of the component processes Xt and Yt is, by itself, a Markov chain on X with transition probabilities p. Moreover, if the transition probability kernel p is ergodic (irreducible and aperiodic) and the state space X is finite, then with probability one (Xt , Yt ) will eventually be absorbed in the diagonal, that is, eventually Xt = Yt . PROOF: It is easy to see that marginally the sequence Xt is a Markov chain with transition probabilities p(x, x0) by summing over the variable y0 in (21) above. It is also obvious that the diagonal is an absorbing set for the Markov chain (Xt , Yt ). Thus, what remains is to prove that the diagonal is accessible from every state (x, y) ? X × X . For this, it is enough to show that there is a positive-probability path from (x, y) to the diagonal. Here we use the hypothesis that the transition kernel p is aperiodic and irreducible: this assures that for some integer n ? 1 the n?step transition probability matrix Pn has all entries positive. In particular, there exists z ? X such that both pn (x, z) > 0 and pn (y, z) > 0. But this implies that there are sequences of states x = x1, x2 , . . . , xn = z and y = y1 , y2, . . . , yn = z such that p(xi , xi+1 ) > 0 and p(yi , yi+1 ) > 0 for each i. Then q((xi , yi ), (xi+1 , yi+1 )) > 0 for each i t}. Example:2 Consider a two-state Markov chain with the state space {a, b, c}. If the Markov chain is in state a, it switches from the current state to one of the other two states, each with probability 1/4, or remains in the same state If it is in state b, then it switches to state c with probability 1/2 or remains the same state. If it is in state c, it switches to state a with probability 1 p= 1/2 1/4 1/4 0 1/2 1/2 1 0 0 This Markov chain is irreducible since it can go from any state to any other state in finite time with non-zero probability. Next, note that there is a non-zero probability of remaining in state a if the Markov chain starts in state a. Therefore, P (n) aa > 0 for all n and state a is aperiodic. Since the Markov chain is irreducible, this implies that all the states are aperiodic. Thus, the finite-state Markov chain is irreducible and aperiodic, which implies the existence of a stationary distribution to which the probability distribution converges starts from any initial distribution. To compute the stationary distribution p, we solve the equation = P where = (a b c), subject to the constraint a + b + c = 1 and a, b, c 0 to obtain = (1/2 1/4 1/4). Answer 3.0 Let us first remember that a time-homogeneous Markov chain at time n is characterized by its distribution (n) = ( (n) i , i S), where (n) i = P(Xn = i), and that (n+1) = (n)P, i.e. (n+1) j = X iS (n) i pij , j S A distribution = ( i , i S) is said to be a stationary distribution for the Markov chain (Xn, n 0) if = P, i.e. j = X iS i pij , j S Remarks. * does not necessarily exist, nor is it necessarily unique * As we will see, if exists and is unique, then i can always be interpreted as the average proportion of time spent by the chain X in state i. It also turns out in this case that E(Ti/X0 = i) = 1/ where Ti = inf{n 0 : Xn = i} is the first time the chain comes back to state i. * If (0) = , then (1) = P = ; likewise, (n) = P n = . . . = , n 0, that is, if the initial distribution of the chain is stationary (we also say the chain is “in stationary state” by abuse of language), then it remains stationary over time. If a chain has more than one closed communicating class, its stationary distributions will not be unique (consider any closed communicating class Ci in the chain; each one will have its own unique stationary distribution i . Extending these distributions to the overall chain, setting all values to zero outside the communication class, yields that the set of invariant measures of the original chain is the set of all convex combinations of the i