Continuous-time Markov chains

Let $\lambda \mu > 0$ and let $X$ be a Markov chain on $\left\{ 1,2\right\}$ with generator

As a continuation of the previous exercise, find:

Jobs arrive in a computer queue in the manner of a Poisson process with intensity $\lambda$. The central processor handles them one by one in the order of their arrival, and each has an exponentially distributed runtime with parameter $\mu$, the runtimes of different jobs being independent of each other and of the arrival process. Let $X(t)$ be the number of jobs in the system (either running or waiting) at time $t$, where $X(0) = 0$. Explain why $X$ is a Markov chain, and write down its generator. Show that a stationary distribution exists if and only if $\lambda < \mu$, and find it in this case.

Let $X = \left\{ X(t): t \geq 0\right\}$ be a Markov chain having stationary distribution $\mathbf{\pi }$. We may sample $X$ at the times of a Poisson process: let $N$ be a Poisson process with intensity $\lambda$, independent of $X$, and define $Y_{n} = X\left(T_{n}\right)$. Show that $Y = \left\{ Y_{n}: n \geq 0\right\}$ is a discrete-time Markov chain with the same stationary distribution as $X$. (This exemplifies the 'Pasta' property: Poisson arrivals see time averages.) [The full assumption of the independence of $N$ and $X$ is not necessary for the conclusion. It suffices that $\left\{ N(s): s \geq t\right\}$ be independent of $\left\{ X(s): s \leq t\right\}$, a property known as 'lack of anticipation'. It is not even necessary that $X$ be Markov; the Pasta property holds for many suitable ergodic processes.]

Let $X$ be a continuous-time Markov chain with generator $\mathbf{G}$ satisfying $g_{i} = -g_{i i} > 0$ for all $i$. Let $H_{A} = \inf \left\{ t \geq 0: X(t) \in A\right\}$ be the hitting time of the set $A$ of states, and let $\eta_{j} = \mathbb {P}_{j}\left(H_{A} < \infty \right)$ be the chance of ever reaching $A$ from $j$. By using properties of the jump chain, which you may assume to be well behaved, show that $\sum_{k} g_{j k} \eta_{k} = 0$ for $j \notin A$.

In continuation of the preceding exercise, let $\mu_{j} = \mathbb {E}_{j}\left[H_{A}\right]$. Show that the vector $\mathbf{\mu }$ is the minimal non-negative solution of the equations

Let $X$ be a continuous-time Markov chain with transition probabilities $p_{i j}(t)$ and define $F_{i} = \inf \left\{ t > T_{1}: X(t) = i\right\}$ where $T_{1}$ is the time of the first jump of $X$. Show that, if $g_{i i} \neq 0$, then $\mathbb {P}_{i}\left(F_{i} < \infty \right) = 1$ if and only if $i$ is recurrent.

Let $X$ be the simple symmetric random walk on the integers in continuous time, so that

Show that the walk is recurrent. Let $T$ be the time spent visiting $m$ during an excursion from 0. Find the distribution of $T$.

Let $i$ be a transient state of a continuous-time Markov chain $X$ with $X(0) = i$. Show that the total time spent in state $i$ has an exponential distribution.

Let $X$ be an asymmetric simple random walk in continuous time on the non-negative integers with retention at $0$, so that

Suppose that $X(0) = 0$ and $\lambda > \mu$. Show that the total time $V_{r}$ spent in state $r$ is exponentially distributed with parameter $\lambda -\mu$.

Assume now that $X(0)$ has some general distribution with probability generating function $G$. Find the expected amount of time spent at 0 in terms of $G$.

Let $X$ be the continuous-time Markov chain on the finite state space $S$ with generator $\mathbf{G} = \left(g_{i j}\right)$. Show from first principles that the transition probabilities satisfy

Let $X$ be a Markov chain on the integers $\mathbb {Z}$ with generator satisfying $g_{i, i-1} = g_{i, i+1} = 2^{i}$ for $i \in \mathbb {Z}$, and $g_{i, j} = 0$ for other pairs $(i, j)$ with $i \neq j$. Does $X$ explode?

A population grows subject to the threat of total annihilation. It is modelled as a Markov chain with generator $\mathbf{G} = \left(g_{i j}\right)$ satisfying

the other off-diagonal elements of $\mathbf{G}$ being 0. Show that the chain is null recurrent.

Let $Z_{n} = X(n h)$ where $X$ is a Markov chain and $h > 0$. Show that $i$ is recurrent for $Z$ if and only if it is recurrent for $X$. Show that $Z$ is irreducible if and only if $X$ is irreducible.