A state in a Markov chain is periodic if the chain can return to the state only at multiples of some integer larger than 1. Periodic behavior complicates the study of the limiting behavior of the chain. As we will see in this section, we can eliminate the periodic behavior by considering the $d$-step chain, where $d$ is the period, but only at the expense of introducing additional equivalence classes. Thus, in a sense, we can trade one form of complexity for another.

As usual, our starting point is a (time homogeneous) Markov chain $X=\left(\begin{array}{c}X_{0}\\ X_{1}\\ X_{2}\\ \end{array}\right)$ with countable state space $S$ and transition probability matrix $P$. The period of state $x\in S$ is

$$d(x)=\gcd (n\in , P^{n}(x, x)> 0)$$Thus, starting in $x$, the chain can return to $x$ only at multiples of the period $d$, and $d$ is the largest such integer. State $x$ is aperiodic if $d(x)=1$ and periodic if $d(x)> 1$. Perhaps the most important result is that period, like recurrence and transience, is a class property.

Show that if $(x, y)$ then $d(x)=d(y)$.

- Suppose that $(x, y)$. The result is trivial if $x=y$, so let's assume that $x\neq y$
- Recall that there exist positive integers $j$ and $k$ such that $P^{j}(x, y)> 0$ and $P^{k}(y, x)> 0$. Show that $P^{(j+k)}(x, x)> 0$ and hence $d(x) | (j+k)$
- Suppose that $n$ is a positive integer with $P^{n}(y, y)> 0$. Show that $P^{(j+k+n)}(x, x)> 0$ and hence $d(x) | (j+k+n)$
- From (b) and (c), conclude that $d(x) | n$
- Now conclude from the definition of period that $d(y) | d(x)$
- Reverse the roles of $x$ and $y$ to conclude that $d(x) | d(y)$
- Conclude that $d(x)=d(y)$

Thus, the definitions of period, periodic, and aperiodic apply to equivalence classes as well as individual states. When the chain is irreducible, we can apply these terms to the entire chain.

Suppose that $x\in S$.

- Show that if $P(x, x)> 0$ then $x$ (and hence the equivalence class of $x$) is aperiodic.
- Sketch a state diagram to show that the converse of (a) is not true.

Suppose now that $X=\left(\begin{array}{c}X_{0}\\ X_{1}\\ X_{2}\\ \end{array}\right)$ is irreducible and is periodic with period $d$. There is no real loss in generality in assuming that the chain is irreducible, for if this were not the case, we could simply restrict our attention to one of the closed, irreducible equivalence classes. Now, we fix a reference state $u\in S$, and for $k\in \{0, 1, , d-1\}$, define

$$A_{k}=\{x\in S\colon \exists n\in \mathbb{N}\colon P^{(nd+k)}(u, x)> 0\}$$Suppose that $j$ and $k$ are indices in $\{0, 1, , d-1\}$. Show that $x\in A_{j}$ and $y\in A_{k}$ if and only if $P^{n}(x, y)> 0$ for some $n=(k-j)\mod d$.

Show that
$\left(\begin{array}{c}A_{0}\\ A_{1}\\ \\ A_{d-1}\end{array}\right)$
are the equivalence classes for the
$d$-step *to and from* relation
$\underset{d}{}$
that governs the
$d$-step chain
$\left(\begin{array}{c}X_{0}\\ X_{d}\\ X_{2d}\\ \end{array}\right)$
that has transition matrix
$P^{d}$

In particular, $\left(\begin{array}{c}A_{0}\\ A_{1}\\ \\ A_{d-1}\end{array}\right)$ partitions the state space $S$; these sets are known as the cyclic classes. The basic structure of the chain is shown in the state diagram below:

Consider the Markov chain with state space $S=\{1, 2, 3, 4, 5, 6, 7\}$ and transition matrix $P$ given below:

$$P=\begin{pmatrix}0 & 0 & 1/2 & 1/4 & 1/4 & 0 & 0\\ 0 & 0 & 1/3 & 0 & 2/3 & 0 & 0\\ 0 & 0 & 0 & 0 & 0 & 1/3 & 2/3\\ 0 & 0 & 0 & 0 & 0 & 1/2 & 1/2\\ 0 & 0 & 0 & 0 & 0 & 3/4 & 1/4\\ 1/2 & 1/2 & 0 & 0 & 0 & 0 & 0\\ 1/4 & 3/4 & 0 & 0 & 0 & 0 & 0\end{pmatrix}$$- Sketch the state digraph and show that the chain is irreducible.
- Find the period $d$.
- Find $P^{d}$.
- Identify the cyclic classes.

Review the definition of the basic Ehrenfest chain. Show that this chain has period 2, and find the cyclic classes.

Review the definition of the modified Ehrenfest chain. Show that this chain is aperiodic.

Review the definition of the simple random walk on $\mathbb{Z}^{k}$ . Show that the chain is periodic with period 2, and find the cyclic classes.