Jump Markov Processes

Prove that the Poisson process with intensity $\lambda$ is a homogeneous jump Markov process. Find the transition function of this process.

Prove that the transition probabilities of a homogeneous jump Markov process satisfy the Kolmogorov--Chapman equation:

Prove that, in the differentiable case, the transition probabilities of a homogeneous jump Markov process with finitely many states satisfy the forward Kolmogorov equations

as well as the backward Kolmogorov equations

Let a homogeneous jump Markov process have two states 0 and 1, where, say, 0 means that some subject is healthy, and 1 that they are ill. Find the transition probabilities, if the transition intensity matrix equals

for some $\lambda , \mu > 0$. Find the limit of the transition probabilities $p_{ij}(t)$ as $t \to \infty$.

Let $\xi (t)$ be a homogeneous jump Markov process with transition intensity matrix $Q = \left\| q_{ij}\right\|$. Prove that if at time $t$ the process $\xi (\cdot )$ is in state $x_{i}$, then the distribution of the time until the next jump of the process is exponential with parameter $q_{ii}$, independently of the history of the process up to time $t$, and at the moment of the next jump the process moves to state $x_{j} \neq x_{i}$ with probability $-q_{ij}/q_{ii}$, independently of the jump time and of the history of the process before it.

Under the conditions of the previous problem, set $T_{0} = 0$, and let $T_{n}$ be the time of the $n$-th jump of the process $\xi (\cdot )$. Prove that, given $\xi (T_{n}) = x_{i}$, the quantity $T_{n+1} - T_{n}$ has an exponential distribution with parameter $q_{ii}$ and does not depend on the history of the process up to time $T_{n}$.

Prove that, under the conditions of the previous problem, the sequence $\xi_{n}^{*} = \xi (T_{n})$ is a Markov chain. Find the transition probabilities of this embedded Markov chain.

Prove that, for the embedded Markov chain $\xi_{n}^{*}$, the states $x_{i}$ and $x_{j}$ communicate if and only if $p_{ij}(t) > 0$ for all $t > 0$.

Let a jump Markov process with values 1, 2, 3, 4, and 5 be defined by the transition intensity matrix

Find the transition probability matrix and classify the states.

Some particle moves among three states as a homogeneous jump Markov process with transition intensity matrix

Find the transition probabilities over time $t$ and their limits as $t \to \infty$.

A homogeneous jump Markov process with three values is given by the transition intensity matrix

A homogeneous jump Markov process with values 1, 2, 3, 4, 5, and 6 is given by the transition intensity matrix

The dynamics of a system is given by a jump Markov process with transition intensities

Show that all states of the process communicate, and find its stationary distribution.

Let $\xi (t)$ be an integer-valued continuous-time random walk with a reflecting barrier at state 0, i.e., a jump Markov process with transition intensities $q_{i,i+1} = \lambda > 0$ for $i \geq 0$, $q_{i,i-1} = \mu > 0$ for $i \geq 1$, $q_{ii} = -(\lambda +\mu )$ for $i \geq 1$, $q_{00} = -\lambda$, $q_{ij} = 0$ for all other $j \neq i$. Show that a stationary distribution of the process $\xi (t)$ exists if and only if $\lambda < \mu$, and find it in this case.

Let $\xi (t)$ be a continuous-time birth-and-death process on the state space $\mathbb {Z}^{+}$, i.e., a homogeneous jump Markov process with transition intensities $q_{i,i+1} = \lambda_{i}$ for $i \geq 0$ (birth probability in state $i$), $q_{i,i-1} = \mu_{i}$ for $i \geq 1$ (death probability), $q_{ii} = -(\lambda +\mu )_{i}$ for $i \geq 1$, $q_{00} = 0$, $q_{ij} = 0$ for all other $j \neq i$. Here $\lambda$ can be understood as the intensity with which one individual gives birth to offspring, and $\mu$ as the intensity with which one individual dies. Using the embedded Markov chain, show that 0 is an absorbing state of the process, and find the probability of absorption at zero for the process starting from an arbitrary state $i > 0$.

At a factory there are $M$ machines, and each machine working at time $t$ can break down in the time interval $(t, t+h)$ with probability $\mu h + o(h)$ -- independently of what happened before time $t$. An engineer repairs each broken machine. If at time $t$ there is at least one broken machine, then the probability that exactly one of them is repaired in the time interval $(t, t+h)$ is $\lambda h + o(h)$ -- again independently of the history of the process up to time $t$ and of the working machines. Let $\xi (t)$ be the number of machines in working order at time $t$.

A homogeneous jump Markov process $\xi (t)$ with values 1, 2, 3, 4, 5, and 6 is given by the transition intensity matrix