Chapter 1

Vectors of Random Variables

[18/89%]

Show

LaTeX

Show LaTeX

Problem 1.1

If $X$ and $Y$ are random variables, prove that

\operatorname {Var}\left[X\right] = \mathbb {E}_{Y}\left[\operatorname {Var}\left[X \mid Y\right]\right]+\operatorname {Var}_{Y}\left[\mathbb {E}\left[X \mid Y\right]\right]

Generalize this result to vectors $\mathbf{X}$ and $\mathbf{Y}$ of random variables.

Problem 1.2

(a)(b)

Let $\mathbf{X} = \left(X_{1}, X_{2}, X_{3}\right)^{T}$ with

\operatorname {Var}\left[\mathbf{X}\right] =\left[\begin{smallmatrix} 5 & 2 & 3 \\ 2 & 3 & 0 \\ 3 & 0 & 3 \end{smallmatrix}\right]

(a)

Find the variance of $X_{1}-2 X_{2}+X_{3}$ .

(b)

Find the variance matrix of $\mathbf{Y} = \left(Y_{1}, Y_{2}\right)^{T}$ , where $Y_{1} = X_{1}+X_{2}$ and $Y_{2} = X_{1}+X_{2}+X_{3}$ .

Problem 1.3

Let $X_{1}, X_{2}, \ldots , X_{n}$ be random variables with a common mean $\mu$ . Suppose that $\operatorname {Cov}\left[ X_{i}, X_{j} \right] = 0$ for all $i$ and $j$ such that $j > i+1$ . If

Q_{\mathrm{i}} = \sum _{i = 1}^{n}\left(X_{i}-\bar{X}\right)^{2}

and

Q_{2} = \left(X_{1}-X_{2}\right)^{2}+\left(X_{2}-X_{3}\right)^{2}+\cdots +\left(X_{n-1}-X_{n}\right)^{2}+\left(X_{n}-X_{1}\right)^{2}

prove that

E\left[\frac{3 Q_{1}-Q_{2}}{n(n-3)}\right] = \operatorname {Var}\left[\bar{X}\right]

Problem 1.4

Given a random sample $X_{1}, X_{2}, X_{3}$ from the distribution with density function

f(x) = \frac{1}{2} \quad (-1 \leq x \leq 1)

find the variance of $\left(X_{1}-X_{2}\right)^{2}+\left(X_{2}-X_{3}\right)^{2}+\left(X_{3}-X_{1}\right)^{2}$ .

Problem 1.5

If $X_{1}, \ldots , X_{n}$ are independently and identically distributed as $N\left(0, \sigma^{2}\right)$ , and $\mathbf{A}$ and $\mathbf{B}$ are any $n \times n$ symmetric matrices, prove that

\operatorname {Cov}\left[ \mathbf{X}^{T} \mathbf{A X}, \mathbf{X}^{T} \mathbf{B} \mathbf{X} \right] = 2 \sigma ^{4} \operatorname {tr}(\mathbf{A B})

Section 1a1.1 Notation, 1.2 Statistical Models, 1.3 Linear Regression Models, 1.4 Expectation and Covariance Operators[4/75%]

Problem 1a.1

Prove that if a is a vector of constants with the same dimension as the random vector $\mathbf{X}$ , then

\mathbb {E}\left[(\mathbf{X}-\mathbf{a})(\mathbf{X}-\mathbf{a})^{T}\right] = \operatorname {Var}\left[\mathbf{X}\right]+(\mathbb {E}\left[\mathbf{X}\right]-\mathbf{a})(\mathbb {E}\left[\mathbf{X}\right]-\mathbf{a})^{T}

If $\operatorname {Var}\left[\mathbf{X}\right] = \mathbf{\Sigma } = \left(\sigma_{i j}\right)$ , deduce that

\mathbb {E}\left[\left\| \mathbf{X}-\mathbf{a}\right\| ^{2}\right] = \sum _{i} \sigma _{i i}+\left\| \mathbb {E}\left[\mathbf{X}\right]-\mathbf{a}\right\| ^{2}

Problem 1a.2

If $\mathbf{X}$ and $\mathbf{Y}$ are $m \times 1$ and $n \times 1$ vectors of random variables, and $\mathbf{a}$ and b are $m \times 1$ and $n \times 1$ vectors of constants, prove that

\operatorname {Cov}\left[ \mathbf{X}-\mathbf{a}, \mathbf{Y}-\mathbf{b} \right] = \operatorname {Cov}\left[ \mathbf{X}, \mathbf{Y} \right]

Problem 1a.3

Let $\mathbf{X} = \left(X_{1}, X_{2}, \ldots , X_{n}\right)^{T}$ be a vector of random variables, and let $Y_{1} = X_{1}, Y_{i} = X_{i}-X_{i-1}(i = 2,3, \ldots , n)$ . If the $Y_{i}$ are mutually independent random variables, each with unit variance, find $\operatorname {Var}\left[\mathbf{X}\right]$ .

Problem 1a.4

If $X_{1}, X_{2}, \ldots , X_{n}$ are random variables satisfying $X_{i+1} = \rho X_{i} \quad (i = 1,2, \ldots , n-1$ ), where $\rho$ is a constant, and $\operatorname {Var}\left[X_{1}\right] = \sigma^{2}$ , find $\operatorname {Var}\left[\mathbf{X}\right]$ .

Section 1b1.5 Mean and Variance of Quadratic Forms[5/100%]

Problem 1b.1

Suppose that $X_{1}, X_{2}$ , and $X_{3}$ are random variables with common mean $\mu$ and variance matrix

\operatorname {Var}\left[\mathbf{X}\right] = \sigma ^{2}\left[\begin{smallmatrix} 1 & 0 & 0 \\ 0 & 1 & \frac{1}{4} \\ 0 & \frac{1}{4} & 1 \end{smallmatrix}\right]

Find $\mathbb {E}\left[X_{1}^{2}+2 X_{1} X_{2}-4 X_{2} X_{3}+X_{3}^{2}\right]$ .

Problem 1b.2

If $X_{1}, X_{2}, \ldots , X_{n}$ are independent random variables with common mean $\mu$ and variances $\sigma_{1}^{2}, \sigma_{2}^{2}, \ldots , \sigma_{n}^{2}$ , prove that $\sum_{i}\left(X_{i}-\bar{X}\right)^{2} /[n(n-1)]$ is an unbiased estimate of $\operatorname {Var}\left[\bar{X}\right]$ .

Problem 1b.3

(a)(b)

Suppose that in Exercise 2 the variances are known. Let $\bar{X}_{w} = \sum_{i} w_{i} X_{i}$ be an unbiased estimate of $\mu$ (i.e., $\sum_{i} w_{i} = 1$ ).

(a)

Prove that $\operatorname {Var}\left[\bar{X}_{w}\right]$ is minimized when $w_{i} \propto 1 / \sigma_{i}^{2}$ . Find this minimum variance $v_{\text{min }}$ .

(b)

Let $S_{w}^{2} = \sum_{i} w_{i}\left(X_{i}-\bar{X}_{w}\right)^{2} /(n-1)$ . If $w_{i} \sigma_{i}^{2} = a(i = 1,2, \ldots , n)$ , prove that $\mathbb {E}\left[S_{w}^{2}\right]$ is an unbiased estimate of $v_{\text{min }}$ .

Problem 1b.4

(a)(b)

The random variables $X_{1}, X_{2}, \ldots , X_{n}$ have a common nonzero mean $\mu$ , a common variance $\sigma^{2}$ , and the correlation between any pair of random variables is $\rho$ .

(a)

Find $\operatorname {Var}\left[\bar{X}\right]$ and hence prove that $-1 /(n-1) \leq \rho \leq 1$ .

(b)

Q = a \sum _{i = 1}^{n} X_{i}^{2}+b\left(\sum _{i = 1}^{n} X_{i}\right)^{2}

is an unbiased estimate of $\sigma^{2}$ , find $a$ and $b$ . Hence show that, in this case,

Q = \sum _{i = 1}^{n} \frac{\left(X_{i}-\bar{X}\right)^{2}}{(1-\rho )(n-1)}

Problem 1b.5

(a)(b)(c)

Let $X_{1}, X_{2}, \ldots , X_{n}$ be independently distributed as $N\left(\mu , \sigma^{2}\right)$ . Define

S^{2} = \frac{1}{n-1} \sum _{i = 1}^{n}\left(X_{i}-\bar{X}\right)^{2}

and

Q = \frac{1}{2(n-1)} \sum _{i = 1}^{n-1}\left(X_{i+1}-X_{i}\right)^{2}

(a)

Prove that $\operatorname {Var}\left[S^{2}\right] = 2 \sigma^{4} /(n-1)$ .

(b)

Show that $Q$ is an unbiased estimate of $\sigma^{2}$ .

(c)

Find the variance of $Q$ and hence show that as $n \rightarrow \infty$ , the efficiency of $Q$ relative to $S^{2}$ is $\frac{2}{3}$ .

Section 1c1.6 Moment Generating Functions and Independence[4/100%]

Problem 1c.1

If $X$ and $Y$ are random variables with the same variance, prove that $\operatorname {Cov}\left[ X+Y, X-Y \right] = 0$ . Give a counterexample which shows that zero covariance does not necessarily imply independence.

Problem 1c.2

Let $X$ and $Y$ be discrete random variables taking values 0 or 1 only, and let $\operatorname {pr}(X = i, Y = j) = p_{i j}(i = 1,0 ; j = 1,0)$ . Prove that $X$ and $Y$ are independent if and only if $\operatorname {Cov}\left[ X, Y \right] = 0$ .

Problem 1c.3

If $X$ is a random variable with a density function symmetric about zero and having zero mean, prove that $\operatorname {Cov}\left[ X, X^{2} \right] = 0$ .

Problem 1c.4

If $X, Y$ and $Z$ have joint density function

f(x, y, z) = \frac{1}{8}(1+x y z) \quad (-1 \leq x, y, z \leq 1)

prove that they are pairwise independent but not mutually independent.