Classical Least Squares

Hooke's law says that the displacement $y$ of an ideal spring is proportional to the force $x$ that is applied — i.e., $y = kx$ for some constant $k$. Consider a spring in which $k$ is unknown. Various masses are attached, and the resulting displacements shown below are observed. Using these observations, determine the least squares estimate for $k$.

Figure 4.6.6

Show that the slope of the line that passes through the origin in $\mathbb {R}^{2}$ and comes closest in the least squares sense to passing through the points $\left\{ (x_1, y_1), (x_2, y_2), \ldots , (x_n, y_n)\right\}$ is given by $m = \sum_i x_i y_i / \sum_i x_i^2$.

A small company has been in business for three years and has recorded annual profits (in thousands of dollars) as follows.

Assuming that there is a linear trend in the declining profits, predict the year and the month in which the company begins to lose money.

An economist hypothesizes that the change (in dollars) in the price of a loaf of bread is primarily a linear combination of the change in the price of a bushel of wheat and the change in the minimum wage. That is, if $B$ is the change in bread prices, $W$ is the change in wheat prices, and $M$ is the change in the minimum wage, then $B = \alpha W + \beta M$. Suppose that for three consecutive years the change in bread prices, wheat prices, and the minimum wage are as shown below.

Use the theory of least squares to estimate the change in the price of bread in Year 4 if wheat prices and the minimum wage each fall by $1.

Suppose that a researcher hypothesizes that the weight loss of a pint of ice cream during storage is primarily a linear function of time. That is,

where $y =$ the weight loss in grams, $t =$ the storage time in weeks, and $\varepsilon$ is a random error function whose mean value is $0$. Suppose that an experiment is conducted, and the following data is obtained.

After studying a certain type of cancer, a researcher hypothesizes that in the short run the number ($y$) of malignant cells in a particular tissue grows exponentially with time ($t$). That is, $y = \alpha_0 e^{\alpha_1 t}$. Determine least squares estimates for the parameters $\alpha_0$ and $\alpha_1$ from the researcher's observed data given below.

Using least squares techniques, fit the following data

with a line $y = \alpha_0 + \alpha_1 x$ and then fit the data with a quadratic $y = \alpha_0 + \alpha_1 x + \alpha_2 x^2$. Determine which of these two curves best fits the data by computing the sum of the squares of the errors in each case.

Consider the time ($T$) it takes for a runner to complete a marathon (26 miles and 385 yards). Many factors such as height, weight, age, previous training, etc.can influence an athlete's performance, but experience has shown that the following three factors are particularly important:

A linear model hypothesizes that the time $T$ (in minutes) is given by $T = \alpha_0 + \alpha_1 x_1 + \alpha_2 x_2 + \alpha_3 x_3 + \varepsilon$, where $\varepsilon$ is a random function accounting for all other factors and whose mean value is assumed to be zero. On the basis of the five observations given below, estimate the expected marathon time for a 43-year-old runner of height 74 in., weight 180 lbs., who has run 450 miles during the previous eight weeks.

What is your personal predicted mean marathon time?

For $\mathbf{A} \in \mathbb {R}^{m \times n}$ and $\overrightarrow {b} \in \mathbb {R}^{m}$, prove that $\overrightarrow {x}_2$ is a least squares solution for $\mathbf{A}\overrightarrow {x} = \overrightarrow {b}$ if and only if $\overrightarrow {x}_2$ is part of a solution to the larger system

In many least squares applications, the underlying data matrix $\mathbf{A}_{m \times n}$ does not have independent columns — i.e., $\operatorname {rk}\left(\mathbf{A}\right) < n$ — so the corresponding system of normal equations $\mathbf{A}^T \mathbf{A}\overrightarrow {x} = \mathbf{A}^T \overrightarrow {b}$ will fail to have a unique solution. This means that in an associated linear estimation problem of the form

there will be infinitely many least squares estimates for the parameters $\alpha_i$, and hence there will be infinitely many estimates for the mean value of $y$ at any given point $(t_1, t_2, \ldots , t_n)$ — which is clearly undesirable. In order to remedy this problem, we restrict ourselves to making estimates only at those points $(t_1, t_2, \ldots , t_n)$ that are in the row space of $\mathbf{A}$. If

is any least squares solution (i.e., $\mathbf{A}^T \mathbf{A}\overrightarrow {x} = \mathbf{A}^T \overrightarrow {b}$), prove that the estimate defined by

is unique in the sense that $\hat{y}$ is independent of which least squares solution $\overrightarrow {x}$ is used.