Markov Processes. Discrete and Continuous Time

Prove that definition (1) is equivalent to the following: for each $s \in T$ and any $A \in \mathscr {F}_{s}, B \in \mathscr {F}_{ \geqslant s}$ the following equality holds

Definition 1: a process $X$ adapted to the filtration $\left(\mathscr {F}_{t}\right)_{t \in T}$ is called Markov (with respect to this filtration and the measure $\mathbb {P}$) if for each $s \in T$ and any event $C \in \mathscr {F}_{ \geqslant s} = \sigma \left\{ X_{u}, u \geqslant s, u \in T\right\}$

Let $X = \left\{ X_{t}, t \in T\right\}$ be a Markov process, $T \subset \mathbb {R}$. Explain why $\left\{ X_{t}, t \in U\right\}$, where $U \subset T$, is also a Markov process. In particular, if $\left\{ X_{t}, t \geqslant 0\right\}$ is a Markov process, then for any $\Delta > 0$ the process $X^{\Delta } = \left\{ X_{k \Delta }, k \geqslant 0\right\}$ is Markov. Is the converse true?

Let $X = \left\{ X_{t}, t \geqslant 0\right\}$ be a Markov process taking values, for each $t$, in a measurable space $(\mathrm{S}, \mathscr {B})$. Let $(\mathrm{V}, \mathscr {E})$ be a measurable space and $h_{t}: \mathrm{S} \rightarrow \mathrm{V}, h_{t} \in \mathscr {B} \mid \mathscr {E}, t \geqslant 0$. Prove that if $h_{t}$ is a one-to-one mapping for each $t \geqslant 0$, then $Y = \left\{ Y_{t} = h_{t}\left(X_{t}\right), t \geqslant 0\right\}$ is also a Markov process. Give an example showing that if the condition that the mapping $h_{t}$ be one-to-one is dropped, this assertion may fail.

Let $X = \left\{ X_{t}, t \geqslant 0\right\}$ be a Markov process with values in $S \subset \mathbb {R}$. Set $Y_{t} = \left[X_{t}\right]$, where $\left[\cdot \right]$ is the integer part of a number. Can one assert that the process $Y = \left\{ Y_{t}, t \geqslant 0\right\}$ is Markov?

(compare with Exercise 3). Let $W = \left\{ W(t) = \left(W_{1}(t), \ldots , W_{m}(t)\right), t \geqslant 0\right\}$ be an $m$-dimensional Brownian motion and $X_{m} = \left\{ X_{m}(t) = \left(\sum_{k = 1}^{m} W_{k}^{2}(t)\right)^{1 / 2}, t \geqslant 0\right\}$ the Bessel process, i.e., the radial part of the process $W$. Is the process $X_{m}$ Markov? Note that $X_{1}(t) = \left|W_{1}(t)\right|$ for $t \geqslant 0, m = 1$.

Let $X = \left\{ X_{t}, t \geqslant 0\right\}$ and $Y = \left\{ Y_{t}, t \geqslant 0\right\}$ be real Markov processes. Are the processes $X+Y = \left\{ X_{t}+Y_{t}, t \geqslant 0\right\}$ and $X Y = \left\{ X_{t} Y_{t}, t \geqslant 0\right\}$ Markov? What can be said about the special case where $Y_{t} = c(t)$, with $c(t)$ a deterministic function?

Let $\left\{ X_{k}, k = 0,1, \ldots \right\}$ be a real Markov process. Set $X_{t} = (t-k) X_{k}+(k+1-t) X_{k+1}$ for $t \in [k, k+1), k = 0,1, \ldots$, i.e., consider the continuous polygonal line on $[0, \infty )$ with vertices $\left(k, X_{k}\right)$. Is the process $X = \left\{ X_{t}, t \geqslant 0\right\}$ Markov? Is the process $Y = \left\{ Y_{t} = X_{\left[t\right]}, t \geqslant 0\right\}$, where $\left[\cdot \right]$ is the integer part of a number, Markov?

Let $\xi_{1}, \xi_{2}, \ldots$ be independent identically distributed random variables taking the values 1 and -1 with probability $1 / 2$ each. Set $S_{0} = 0$, $S_{n} = \xi_{1}+\cdots +\xi_{n}, n \geqslant 1$, and $X_{n} = \max_{0 \leqslant k \leqslant n} S_{k}$. Prove that the process $X = \left\{ X_{n}, n \geqslant 0\right\}$ is not Markov.

Give an example of a martingale that is not a Markov process. Construct an example of a real Markov process $X = \left\{ X_{t}, t \in T\right\}$ with $\mathbb {E}\left[\left|X_{t}\right|\right] < \infty$ for all $t \in T$ that is not a martingale.

Let $\eta , \xi_{1}, \xi_{2}, \ldots$ be independent variables, where $\xi_{1}, \xi_{2}, \ldots$ are uniformly distributed on $[0,1]$ and $\eta$ is a real random variable with distribution function $F_{\eta }(x)$. Set $S_{n} = 1$ if $\xi_{n} \leqslant \eta$, and $S_{n} = -1$ if $\xi_{n} > \eta (n = 1,2, \ldots )$. Is $\left\{ S_{n}, n \geqslant 1\right\}$ a homogeneous Markov process?

Is it true that a real Gaussian process $X = \left\{ X_{t}, t \geqslant 0\right\}$ is Markov if condition (12) holds only for the function $f(x) \equiv x, x \in \mathbb {R}$ ?

Theorem 2. A process $X = \left\{ X_{t}, t \in T\right\}$ taking, for each $t \in T$, values in a Borel space $\left(\mathrm{S}_{t}, \mathscr {B}_{t}\right)$ is Markov if and only if it has the following property: for any $m \geqslant 1$ and all $s_{1} < \cdots < s_{m} < s \leqslant t$ ($s, t$ and all $s_{i}$ in $T$), and for an arbitrary bounded $\mathscr {B}_{t}$-measurable function $f$

Prove that a real Gaussian process $X = \left\{ X_{t}, t \in T\right\}$, $T \subset \mathbb {R}$, is Markov if and only if for any $t_{1} < t_{2} < t_{3} \left(t_{1}, t_{2}, t_{3} \in T\right)$ the following equality holds

where $r(s, t) = \operatorname {cov}\left(X_{s}, X_{t}\right), s, t \in T$. Is fractional Brownian motion a Markov process?

Is the Ornstein-Uhlenbeck process (Definition 13 of Chapter III) a homogeneous Markov process?

Definition 13 of Chapter III. The Ornstein-Uhlenbeck process with parameters $\alpha , \beta > 0$ is the process given by the formula

where $W = \left\{ W(t), t \geqslant 0\right\}$ is a standard Brownian motion.

Let $\mathbb {P}\left(X_{0} = 1\right) = \mathbb {P}\left(X_{0} = -1\right) = 1 / 2$, and let $X_{0}$ be independent of the Poisson process $N = \left\{ N_{t}, t \geqslant 0\right\}$ with intensity $\lambda > 0$. Let the process $X = \left\{ X_{t}, t \geqslant 0\right\}$ change its value to the value of the opposite sign at the jump points of the Poisson process. Is the process $X$, called the telegraph signal, Markov? Draw the graph of a sample path of this process.

For $x \geqslant 0$ let $\tau_{x} = \inf \left\{ t \geqslant 0: W(t) = x\right\}$, where $W = \left\{ W(t), t \geqslant 0\right\}$ is a Wiener process. Prove that the process $\left\{ \tau_{x}, x \geqslant 0\right\}$ is a process with independent increments (hence Markov, by Theorem 3).

Theorem 3. Let $X = \left\{ X_{t}, t \in T\right\}$ be a process with independent increments taking values in $\mathbb {R}^{d}$, $d \geqslant 1$. Then $X$ is a Markov process (with respect to the natural filtration).

Let $X = \left\{ X_{t}, t \geqslant 0\right\}$ be a Markov process. For $s > 0$ define the process $Y = \left\{ Y_{t} = X_{s-t}, t \in [0, s]\right\}$. Will $Y$ be a Markov process? Must the process $Y$ be a homogeneous process if $X$ is one?

Prove that the Brownian bridge (see the definition in $\S 9$ of Chapter V) is an inhomogeneous Markov process.

Definition from $\S 9$ of Chapter V: $W^{0}$ is the Brownian bridge, defined as the process $W^{0}(t) = \left\{ W(t)-t W(1), t \in [0,1]\right\}$, where $W = \left\{ W(t), t \in [0,1]\right\}$ is a Wiener process on the interval $[0,1]$.

Let $X = \left\{ X_{t}, t \geqslant 0\right\}$ be a stochastic process on $[0, \infty )$, and

Prove that the processes $X$ and $Y$ can only be Markov simultaneously. Prove that if $Y$ is a Markov process, then this process is necessarily homogeneous. How are the transition functions of the Markov processes $X$ and $Y$ related?

Let $X = \left\{ X_{t}, t \geqslant 0\right\}$ be a homogeneous Markov process with values (for each $t$) in some Polish space $\mathrm{S}$. Prove that there exist measurable functions $h_{s}: \mathrm{S} \times [0,1] \rightarrow \mathrm{S}, s \geqslant 0$ (i.e. $h_{s} \in \mathscr {B}(\mathrm{S}) \otimes \mathscr {B}([0,1]) \mid \mathscr {B}(\mathrm{S}), s \geqslant 0$) and uniformly distributed random variables $\theta_{t, s}$ independent of $\left\{ X_{u}, 0 \leqslant u \leqslant t\right\}$ for $t, s \geqslant 0$, such that $X_{t+s} = h_{s}\left(X_{t}, \theta_{t, s}\right)$ a.s. for all $t, s \geqslant 0$.

Let a homogeneous Markov process take values in $\mathbb {R}^{m}$, and let its transition function be such that $P(x, t, B) = P(x+y, t, B+y)$ for all $x, y \in \mathbb {R}^{m}$, $t \geqslant 0, B \in \mathscr {B}\left(\mathbb {R}^{m}\right)$. Prove that then this process has independent increments.

Prove that for the standard Poisson process with intensity $\lambda > 0$ the "strong law of large numbers" holds: $N(t) / t \rightarrow \lambda$ a.s. as $t \rightarrow \infty$.

("waiting time paradox") Let the Poisson process $N = \left\{ N_{t}, t \geqslant 0\right\}$ be constructed according to Theorem 2 of Chapter II from a sequence $\left\{ \xi_{j}\right\}$ of independent exponential (with parameter $\lambda$) random variables.

Theorem 2 of Chapter II (explicit construction of the Poisson process). Let $\xi_{1}, \xi_{2}, \ldots$ be independent random variables having the exponential distribution with density

where $\lambda$ is a positive parameter, $i \in \mathbb {N}$. Then the corresponding renewal process (I.10) is a Poisson process with intensity $\lambda$.

Formula (I.10) (Example 2 of Chapter I). For a sequence $\xi_{1}, \xi_{2}, \ldots$ of independent identically distributed nondegenerate nonnegative random variables, the renewal process is defined by the formula

For fixed $t > 0$ find the distribution of the random variable $\zeta_{t} = \sum_{j = 1}^{N_{t}+1} \xi_{j}-t$, i.e. of the "overshoot" over the level $t$ of the accumulating partial sums $S_{k} = \sum_{j = 1}^{k} \xi_{j}, k \geqslant 1$. The following amusing question is related to this problem. Let $\xi_{j}$ be the random intervals between the arrival times of buses at a given stop. The question is: how long will you wait for the nearest bus to appear if you happen to be at the stop at time $t$?

(continuation of Exercise 22) Prove that for each fixed $t > 0$ the random variables $\zeta_{t}, \xi_{N_{t}+2}, \xi_{N_{t}+3}, \ldots$ introduced above form a sequence of independent exponentially distributed random variables with parameter $\lambda$.

Define $m(B) = \mathbb {E}\left[\mu (B, \cdot )\right]$ for all $B \in \mathscr {B}\left(\mathbb {R}_{+}\right)$, where $\mu (B, \cdot )$ appears in (85).

Suppose that on some probability space $(\Omega , \mathscr {F}, \mathbb {P})$ we are given a sequence of random variables $\left\{ \tau_{k}\right\}_{k \geqslant 1}$ such that for $\omega \in \Omega_{0}$, where $\mathbb {P}\left(\Omega_{0}\right) = 1$,

Definition 11. The random point measure $\mu : \mathscr {B}\left(\mathbb {R}_{+}\right) \times \Omega \rightarrow \overline{\mathbb {Z}}_{+} = \mathbb {Z}_{+} \cup \left\{ \infty \right\}$ is given by the formula

where the sequence $\left\{ \tau_{k}\right\}_{k \geqslant 1}$ is described above, and $\delta_{x}$ is the Dirac measure.

Will the function $m(\cdot )$ be a measure on $\mathscr {B}\left(\mathbb {R}_{+}\right)$? Can a Poisson process with a locally finite intensity measure $m$ be represented in the form (86)?

Definition 12. The random point process $Y = \left\{ Y(t), t \geqslant 0\right\}$, where $Y(0) = 0$, is introduced for $t > 0$ as follows:

where the random point measure $\mu$ is given by formula (85).

(compare with Theorem 14 of Chapter V). Prove that the random point process (86) is a Poisson process if $Y$ has independent increments and $Y$ is stochastically continuous on $\mathbb {R}_{+}$ (see (I.56)).

Theorem 14 of Chapter V (Doob). Let a process $X = \left\{ X_{t}, t \geqslant 0\right\}$ take values in $\mathbb {R}^{m}$ for each $t$ and have independent increments. If the sample paths of the process are continuous a.s., then the process $Y = \left\{ Y_{t} = X_{t}-X_{0}, t \geqslant 0\right\}$ is Gaussian. Moreover, for $0 \leqslant s \leqslant t < \infty$

where $M: \mathbb {R}_{+} \rightarrow \mathbb {R}^{m}$ and $G: \mathbb {R}_{+} \rightarrow \mathbb {R}^{m^2}$ are continuous functions.

Suppose that on some probability space $(\Omega , \mathscr {F}, \mathbb {P})$ we are given a sequence of random variables $\left\{ \tau_{k}\right\}_{k \geqslant 1}$ such that for $\omega \in \Omega_{0}$, where $\mathbb {P}\left(\Omega_{0}\right) = 1$,

Definition 11. The random point measure $\mu : \mathscr {B}\left(\mathbb {R}_{+}\right) \times \Omega \rightarrow \overline{\mathbb {Z}}_{+} = \mathbb {Z}_{+} \cup \left\{ \infty \right\}$ is given by the formula

where the sequence $\left\{ \tau_{k}\right\}_{k \geqslant 1}$ is described above, and $\delta_{x}$ is the Dirac measure.

Definition 12. The random point process $Y = \left\{ Y(t), t \geqslant 0\right\}$, where $Y(0) = 0$, is introduced for $t > 0$ as follows:

where the random point measure $\mu$ is given by formula (85).

Formula (I.56): a process $X = \left\{ X_{t}, t \in T\right\}$ with values in a metric space $(\mathrm{S}, \rho )$ is stochastically continuous on $T$ if for each $t \in T$ and any $\varepsilon > 0$

By Theorem 3, the random variables $\tau_{k} = \xi_{1}+\cdots +\xi_{k}, k \in \mathbb {N}$, form a Markov chain if the summands are independent real random variables.

Theorem 3. Let $X = \left\{ X_{t}, t \in T\right\}$ be a process with independent increments taking values in $\mathbb {R}^{d}$, $d \geqslant 1$. Then $X$ is a Markov process (with respect to the natural filtration).

Will the process (86) be Markov if $\left\{ \tau_{k}, k \in \mathbb {N}\right\}$ is an arbitrary Markov chain satisfying condition (84)?

Suppose that on some probability space $(\Omega , \mathscr {F}, \mathbb {P})$ we are given a sequence of random variables $\left\{ \tau_{k}\right\}_{k \geqslant 1}$ such that for $\omega \in \Omega_{0}$, where $\mathbb {P}\left(\Omega_{0}\right) = 1$,

Definition 11. The random point measure $\mu : \mathscr {B}\left(\mathbb {R}_{+}\right) \times \Omega \rightarrow \overline{\mathbb {Z}}_{+} = \mathbb {Z}_{+} \cup \left\{ \infty \right\}$ is given by the formula

where the sequence $\left\{ \tau_{k}\right\}_{k \geqslant 1}$ is described above, and $\delta_{x}$ is the Dirac measure.

Definition 12. The random point process $Y = \left\{ Y(t), t \geqslant 0\right\}$, where $Y(0) = 0$, is introduced for $t > 0$ as follows:

where the random point measure $\mu$ is given by formula (85).

Definition 13. Suppose that on some probability space $(\Omega , \mathscr {F}, \mathbb {P})$ we are given a Poisson process $N = \left\{ N(t), t \geqslant 0\right\}$ with intensity $\lambda = 1$ and a process $\Lambda = \left\{ \Lambda (t), t \geqslant 0\right\}$ independent of it, with a.s. nondecreasing, finite and right-continuous sample paths starting from zero. The Cox process is the process $Z = \left\{ Z(t), t \geqslant 0\right\}$, where $Z(t, \omega ) = N(\Lambda (t, \omega ), \omega ), t \geqslant 0, \omega \in \Omega$.

Is the Cox process Markov?

Find the covariance function of the process (87) if the sequence $\left\{ \tau_{k}\right\}_{k \geqslant 1}$ defines a Poisson point process with intensity $\lambda$.

Suppose that on some probability space $(\Omega , \mathscr {F}, \mathbb {P})$ we are given a sequence of random variables $\left\{ \tau_{k}\right\}_{k \geqslant 1}$ such that for $\omega \in \Omega_{0}$, where $\mathbb {P}\left(\Omega_{0}\right) = 1$,

Definition 14. We say that a stochastic process $X = \left\{ X(t), t \geqslant 0\right\}$ is "shot noise" if

where the bounded Borel function $\phi : \mathbb {R} \rightarrow \mathbb {R}$ has compact support, and the random variables $\tau_{k}$ ($k \in \mathbb {N}$) satisfy condition (84) (which ensures that the series (87) converges for each $t \geqslant 0$ on the set $\Omega_{0}$ of probability one).

Is it sufficient, for the convergence of the series (87) with probability one for each $t \geqslant 0$, to require that the bounded function $\phi$ be Lebesgue integrable?

Suppose that on some probability space $(\Omega , \mathscr {F}, \mathbb {P})$ we are given a sequence of random variables $\left\{ \tau_{k}\right\}_{k \geqslant 1}$ such that for $\omega \in \Omega_{0}$, where $\mathbb {P}\left(\Omega_{0}\right) = 1$,

Suppose that, besides the sequence $\left\{ \tau_{k}\right\}_{k \geqslant 1}$ of random variables satisfying condition (84), on the same probability space $(\Omega , \mathscr {F}, \mathbb {P})$ there is some sequence $\left\{ \eta_{k}\right\}_{k \geqslant 1}$ of real random variables "attached" to the times $\tau_{k}, k \geqslant 1$.

Definition 15. A marked random point process is a process (indexed by sets $C$) of the following form

The function

is called the counting function.

Draw the graph of the sample path $Y(t, D), t \geqslant 0$, for fixed $D \in \mathscr {B}(\mathbb {R})$.

Let $N = \left\{ N(t), t \geqslant 0\right\}$ be a Poisson process with mean measure $m = m(\cdot )$, determined by the sequence $\left\{ \tau_{k}\right\}_{k \geqslant 1}$ that specifies the jump points of $N$. Let $\left\{ \eta_{k}\right\}_{k \geqslant 1}$ be a sequence of independent identically distributed real random variables with $\operatorname {Law}\left(\eta_{k}\right) = \widetilde{\mathbb {P}}$. Take $D \in \mathscr {B}(\mathbb {R})$ such that $\widetilde{\mathbb {P}}(D) > 0$. Prove that the "thinned process" $Y(t, D), t \geqslant 0$, defined by formula (88), is a Poisson process with mean measure $m(\cdot ) \widetilde{\mathbb {P}}(D)$.

Definition 11. The random point measure $\mu : \mathscr {B}\left(\mathbb {R}_{+}\right) \times \Omega \rightarrow \overline{\mathbb {Z}}_{+} = \mathbb {Z}_{+} \cup \left\{ \infty \right\}$ is given by the formula

where the sequence $\left\{ \tau_{k}\right\}_{k \geqslant 1}$ is described above, and $\delta_{x}$ is the Dirac measure.

Definition 12. The random point process $Y = \left\{ Y(t), t \geqslant 0\right\}$, where $Y(0) = 0$, is introduced for $t > 0$ as follows:

where the random point measure $\mu$ is given by formula (85).

Formula (88):

where the process $Y$ is given by formula (86).

Let $X = \left\{ X(t), t \geqslant 0\right\}$ be a homogeneous standard (i.e., $p_{i j}(t) \rightarrow \delta_{i j}$ as $t \rightarrow 0+$) Markov chain with a finite or countable state space $\mathrm{S}$ and transition probabilities $p_{i j}(t)$, considered in its separable modification, and let $Q = \left(q_{i j}\right)_{i, j \in \mathrm{S}}$ be its infinitesimal matrix:

Let $q_{i} = \infty$ for a state $i$. Then the probability that the process $X$ stays in this state for a positive time is zero, i.e., having entered this state, the process leaves it instantly.

Let $Q = \left(q_{i j}\right)_{i, j \in \mathrm{S}}$ be the infinitesimal matrix of a homogeneous Markov chain, and $q_{i} = -q_{i i}$.

Definition 10. A chain is called conservative if all entries $q_{i j}$ of the infinitesimal matrix $Q$ are finite and, moreover, for every $i$

Let $\mathrm{S}$ be a finite or countable set, and let $X = \left\{ X(t), t \geqslant 0\right\}$ be a conservative Markov chain with infinitesimal matrix $Q$, where $q_{i} \in (0, \infty )$, $i \in \mathrm{S}$. Let us construct recursively sequences of random variables $\left\{ \eta_{n}\right\}_{n \geqslant 1}$ and $\left\{ \tau_{n}\right\}_{n \geqslant 1}$ with values in $\mathrm{S}$ and $\mathbb {R}_{+} = [0, \infty )$ respectively, and such that $\eta_{1}$ has an arbitrary distribution,

and for $n \geqslant 1, i, j, i_{1}, \ldots , i_{n} \in \mathrm{S}, x_{1}, \ldots , x_{n} \in \mathbb {R}_{+}$ let

Prove that on a suitable probability space $(\Omega , \mathscr {F}, \mathbb {P})$ there exist the sequences of random variables $\left\{ \eta_{n}\right\}_{n \geqslant 1}$ and $\left\{ \tau_{n}\right\}_{n \geqslant 1}$ described above, satisfying conditions (92) and (93). Moreover, $0 = \tau_{0} < \tau_{1}(\omega ) < \tau_{2}(\omega ) < \cdots$ a.s. Can one assert that always $\lim_{n \rightarrow \infty } \tau_{n} = \infty$ a.s.? Prove that the latter certainly holds if $\sup q_{i} < \infty$.

Let $\tau_{0}(\omega ) = 0$ a.s. Using the random variables $\eta_{n}, \tau_{n}(n \geqslant 1)$ appearing in Exercise 32 , and assuming that condition (84) is satisfied, let us introduce the process $Y = \left\{ Y(t), t \geqslant 0\right\}$ by setting

(for $\omega$ such that $\tau_{n-1}(\omega ) = \tau_{n}(\omega )$ for some $n \geqslant 1$, let $Y(t, \omega ) = 0$, $t \geqslant 0)$.

Condition (84): for $\omega \in \Omega_{0}$, where $\mathbb {P}\left(\Omega_{0}\right) = 1$,

Prove that $Y$ is a Markov process having the same finite-dimensional distributions as the process $X$ that determined the matrix $Q$. How should the above construction be modified if some of the quantities $q_{i}$ are equal to zero?

Let $X = \left\{ X(t), t \geqslant 0\right\}$ be a homogeneous standard (i.e., $p_{i j}(t) \rightarrow \delta_{i j}$ as $t \rightarrow 0+$) Markov chain with a finite or countable state space $\mathrm{S}$ and transition probabilities $p_{i j}(t)$, and let $Q = \left(q_{i j}\right)_{i, j \in \mathrm{S}}$ be its infinitesimal matrix:

Prove that $\sup_{i} q_{i} < \infty$ if and only if $p_{i i}(t) \rightarrow 1$ as $t \rightarrow 0+$ uniformly in $i \in \mathrm{S}$ (or, equivalently, $p_{i j}(t) \rightarrow \delta_{i j}$ as $t \rightarrow 0+$ uniformly in $i, j \in \mathrm{S}$ ).

Give an example of transition probabilities for which the standardness condition (49) does not hold.

Definition 9. A homogeneous Markov chain $X = \left\{ X_{t}, t \geqslant 0\right\}$ (or the semigroup $P(t) = \left(p_{i j}(t)\right)$ generated by it) is called standard if $P(t) \rightarrow I$ as $t \rightarrow 0+$, i.e., for all $i, j \in \mathrm{S}$

where, as usual, $\delta_{i j}$ is the Kronecker symbol.

Give an example where property (49) holds but (89) does not.

Condition (89), where $q_{i j} = \left.\frac{d^{+} p_{i j}(t)}{d t}\right|_{t = 0}$ are the entries of the infinitesimal matrix $Q$:

Definition 9. A homogeneous Markov chain $X = \left\{ X_{t}, t \geqslant 0\right\}$ (or the semigroup $P(t) = \left(p_{i j}(t)\right)$ generated by it) is called standard if $P(t) \rightarrow I$ as $t \rightarrow 0+$, i.e., for all $i, j \in \mathrm{S}$

where, as usual, $\delta_{i j}$ is the Kronecker symbol.

Let the state space $\mathrm{S}$ consist of a finite number $N$ of elements, and let the transition matrix $P(t)$ be standard. Prove that then this matrix admits the following representation:

where the $N \times N$ matrix $Q$ satisfies the condition

Conversely, if conditions (95) hold for a matrix $Q$, then formula (94) defines a standard matrix of transition probabilities.

The backward system of Kolmogorov equations for the transition probabilities $p_{i j}(t)$ of a homogeneous Markov chain with infinitesimal matrix $Q = \left(q_{i j}\right)_{i, j \in \mathrm{S}}$ has the form

and the forward system has the form

Equations (65) and (72) can be understood either in the narrow sense (i.e., that all $p_{i j}^{\prime }(t)$ are continuous for $t \geqslant 0$ and the equations under study hold for all $t \geqslant 0$ ), or in the broad sense, i.e., that $p_{i j}(t)$ are absolutely continuous and the equations under consideration hold for almost all (with respect to Lebesgue measure) $t \geqslant 0$.

Prove that for the systems of Kolmogorov equations both interpretations given above (in the narrow and in the broad sense) are equivalent. For the forward equations this exercise should be classed among the very difficult ones.

Let $Q = \left(q_{i j}\right)_{i, j \in \mathrm{S}}$ be a matrix with real entries, where $\mathrm{S}$ is countable, such that

The backward and forward systems of Kolmogorov equations for the matrices $P(t) = \left(p_{i j}(t)\right)_{i, j \in \mathrm{S}}$, $t \geqslant 0$, have, respectively, the form

and

(see also Exercise 34). Prove that if $\sup_{i} q_{i} < \infty$, then both systems of Kolmogorov equations have a unique solution.

Definition 17. A homogeneous Markov process $X = \left\{ X(t), t \geqslant 0\right\}$ with state space $S = \left\{ 0,1, \ldots \right\}$ is called a birth-and-death process if its infinitesimal matrix $Q$ has the following form