Worked examples of OLS point estimates

The variables $x$ and $y$ satisfy the linear equation $y=ax+b$, whose coefficients need to be determined. From a series of equally precise measurements, the values $x_{i}$, $y_{i}$ $(i=1,2,\ldots ,n)$ are obtained for the quantities $x$ and $y$. Using the method of least squares, find the most reliable values of the coefficients $a$ and $b$.

In the plane, $n$ points $M_{i}(x_{i},y_{i})$ $(i=1,2,\ldots ,n)$ are given. In what position must the line $x\cos \alpha +y\sin \alpha -p=0$ be so that the sum of squared deviations of the given points from this line is a minimum?

From the two observations $X_{1} = 2, X_{2} = 4$ with $h_{1} = 1, h_{2} = 2$, construct a realization of the OLS estimate of the unknown parameters in the simple linear regression model. Compute the coefficient of determination.

Using the data of Example 10.6, construct the dependence $X = \theta_{1}+\theta_{2} h$, assuming that:

Estimate the unknown parameters $\alpha , \beta$ in the model of a person's consumption of some product $v = \alpha +\beta \ln w$, where $v$ is the expenditure on the consumer product, $w$ is the weekly income (Table 10.4).

Table 10.4

Estimate the elasticity coefficient $\sigma = \frac{d v}{d w} \cdot \frac{w}{v}$. Assuming that the observation errors have the distribution $\mathcal{N}\left(0 ; \sigma^{2}\right)$, use Fisher's test to test the hypothesis $H_{0}: \beta = 1$ at the significance level 0,05.

Table 14.4 gives the indices of output $Y_{k}$, capital inputs $K_{k}$ and labor inputs $L_{k}$ for $n = 10$ years (USA, 1913-1922).

Table 14.4

The relationship between these variables is assumed to have the form

Find estimates of the parameters $\theta_{1}$ and $\theta_{2}$ by OLS and LAD. Find a robust estimate of the parameter $\sigma^{2} = \operatorname {Var}\left[\varepsilon_{k}\right]$. Analyze the data for outliers and reject them.

In quadrilateral $A B C D$, independent measurements of equal precision of the angles $A B D, D B C, A B C, B C D, C D B$, $B D A, C D A, D A B$ (in degrees) gave the results $50.78$, $30.25,78.29,99.57,50.42,40.59,88.87,89.86$ respectively. Assuming that measurement errors are normally distributed according to the law $\mathscr {N}\left(0, \sigma^{2}\right)$, find the optimal estimators of the angles $\beta_{1}=A B D, \beta_{2}=D B C, \beta_{3}=C D B, \beta_{4}=B D A$ and of the unknown variance $\sigma^{2}$.

Let

$i=0,1, \ldots , n$, where $\beta_{1}, \beta_{2}$ are unknown parameters, and $\varepsilon_{0}, \ldots , \varepsilon_{n}$ are independent random variables distributed according to the law $\mathscr {N}\left(0, \sigma^{2}\right)$. Reduce the problem to a linear model and find the least-squares estimators for $\beta_{1}$ and $\beta_{2}$, as well as an unbiased estimator for $\sigma^{2}$.

Using the linear regression method, construct an approximation of the function $f(x)$ by a cubic polynomial based on the following data:

Let

where $\mathbb {E}\left[\varepsilon_{i}\right] = 0(i = 1,2,3)$. Find the least squares estimates of $\theta$ and $\phi$.

The tension $T$ observed in a nonextensible string required to maintain a body of unknown weight $w$ in equilibrium on a smooth inclined plane of angle $\theta (0 < \theta < \pi / 2)$ is a random variable with mean $\mathbb {E}\left[T\right] = w \sin \theta$. If for $\theta = \theta_{i}(i = 1,2, \ldots , n)$ the corresponding values of $T$ are $T_{i}$ ($i = 1,2, \ldots , n$), find the least squares estimate of $w$.

Let $Y_{i} = \beta_{0}+\beta_{1} x_{i}+\varepsilon_{i}(i = 1,2, \ldots , n)$, where $\mathbb {E}\left[\varepsilon \right] = 0$ and $\operatorname {Var}\left[\varepsilon \right] = \sigma^{2} \mathbf{I}_{n}$. Find the least squares estimates of $\beta_{0}$ and $\beta_{1}$. Prove that they are uncorrelated if and only if $\bar{x} = 0$.

In order to estimate two parameters $\theta$ and $\phi$ it is possible to make observations of three types: (a) the first type have expectation $\theta$, (b) the second type have expectation $\theta +\phi$, and (c) the third type have expectation $\theta -2 \phi$. All observations are subject to uncorrelated errors of mean zero and constant variance. If $m$ observations of type (a), $m$ observations of (b), and $n$ observations of type (c) are made, find the least squares estimates $\widehat{\theta }$ and $\widehat{\phi }$. Prove that these estimates are uncorrelated if $m = 2 n$.

Let

where $\bar{x}_{j} = \sum_{i = 1}^{n} x_{i j} / n, \mathbb {E}\left[\varepsilon \right] = \mathbf{0}$, and $\operatorname {Var}\left[\varepsilon \right] = \sigma^{2} \mathbf{I}_{n}$. If $\widehat{\beta }_{1}$ is the least squares estimate of $\beta_{1}$, show that

where $r$ is the correlation coefficient of the $n$ pairs $\left(x_{i 1}, x_{i 2}\right)$.

Obtain the least squares estimates of the $\mu_{i}$ using $\widehat{\mathbf{\beta }} = \left(\mathbf{X}^{T} \mathbf{X}\right)^{-1} \mathbf{X}^{T} \mathbf{Y}$ directly, where $\mathbf{X}$ is given by (8.2).

Let $Y_{i} = a_{i} \beta_{1}+b_{i} \beta_{2}+\varepsilon_{i}(i = 1,2, \ldots , n)$, where the $a_{i}, b_{i}$ are known and the $\varepsilon_{i}$ are independently and identically distributed as $N\left(0, \sigma^{2}\right)$. Find a necessary and sufficient condition for the least squares estimates of $\beta_{1}$ and $\beta_{2}$ to be independent.

Let

and

where $\mathbb {E}\left[\varepsilon_{i}\right] = 0(i = 1,2,3)$. Find the least squares estimates of $\partial_{1}$ and $\theta_{2}$. If the equations above are augmented to

find the least squares estimate of $\theta_{3}$.

Suppose that $\mathbb {E}\left[Y_{t}\right] = \beta_{0}+\beta_{1} \cos \left(2 \pi k_{1} t / n\right)+\beta_{2} \sin \left(2 \pi k_{2} t / n\right)$, where $t = 1,2, \ldots , n$, and $k_{1}$ and $k_{2}$ are positive integers. Find the least squares estimates of $\beta_{0}, \beta_{1}$, and $\beta_{2}$.

Suppose that

where $\alpha_i$ and $\theta$ are unknown parameters, $t_{ij}$ are known constants, and $\varepsilon_{ij}$ are independent and identically distributed random variables with mean $0$. Find explicit forms for the least squares estimators (LSE's) of $\theta$, $\alpha_i$, $i=1,\ldots ,a$.

Consider the linear model $X=Z\beta +\varepsilon$, where $Z$ is a known $n\times p$ matrix, $\beta$ is a $p$-vector of unknown parameters, and $\varepsilon$ is a random $n$-vector whose components are independent and identically distributed with mean $0$ and variance $\sigma^2$. Let $Z=(Z_1,Z_2)$ and $\beta =(\beta_1,\beta_2)$, where $Z_j$ is $n\times p_j$ and $\beta_j$ is a $p_j$-vector, $j=1,2$. Assume that $(Z_1^\tau Z_1)^{-1}$ and $[Z_2^\tau Z_2-Z_2^\tau Z_1(Z_1^\tau Z_1)^{-1}Z_1^\tau Z_2]^{-1}$ exist.