Data mining, spurious predictors and overfitting

In 1960, Meyer Efroymson, a statistician for Esso Research and Engineering, proposed a stepwise procedure for choosing the explanatory variables in a multiple regression model from a list of candidate variables, one by one, based on which variable has the lowest p -value. One explanatory variable is chosen in the first round; a second explanatory variable is added in the second round, and so on, until there are no remaining candidate variables with p-value less than .05. This procedure was recently tested using 100 observations on 50 candidate variables. The estimated model was then used to make predictions for 100 additional observations. The mean square error of the predictions was 348.20 for the 100 observations used to estimate the model and $2,738.43$ for the 100 new observations. How would you explain this disparity?

What are the problems with these statements?

Collect daily data during two recent years for $\mathrm{Y}=$ the percentage price change in the SP 500 and for 100 explanatory variables, each equal to the daily difference between the high and low temperatures in 100 diverse cities. Use the first year of your data to estimate 100 simple regression models, each using one of your 100 explanatory variables. Identify the five cities with the highest t -values and estimate a multiple regression model using only these five cities as explanatory variables. Finally use the estimated coefficients for the multiple regression model you estimated with the first year's data to predict the daily percentage price changes in the SP 500 during the second year. What is your conclusion?

Collect daily data during two recent years for $\mathrm{Y}=$ the percentage price change in the SP 500 and for 100 explanatory variables, with the daily value of each explanatory variable equal to a randomly generated number. Use the first year of your data to estimate 100 simple regression models, each using one of your 100 explanatory variables. Identify the five variables with the highest t -values and estimate a multiple regression model using only these five explanatory variables. Finally use the estimated coefficients for the multiple regression model you estimated with the first year's data to predict the daily percentage price changes in the SP 500 during the second year. What is your conclusion?

Use a random number generator to create 500 observations for 101 variables. Let the first random variable be Y, the variable you want to predict and let the other 100 variables be the potential explanatory variables that will be used to predict Y. Use the first 250 observations to estimate a multiple regression model using the 100 explanatory variables to predict Y. (Alternatively, you can estimate 5 models with 20 explanatory variables.) Identify the five explanatory variables with the highest t -values. Reestimate your multiple regression model using these 5 explanatory variables and the first 250 observations to predict Y.

Now, use the estimated coefficients for the second multiple regression model that you estimated with the first 250 observations to predict the values of Y for the second 250 observations. What is your conclusion?

Select a 10-year period, such as 2015-2024. Estimate a multiple regression model using the following four explanatory variables, all recorded the year before the Super Bowl, to predict the Super Bowl score differential, AFC team score minus NFC team score, in the first 5 years of your data:

• High temperature in London on June 30

• Dow Jones Industrial Average on June 30

• Average number of letters in the names of Physics Nobel Prize winner(s) that year

• Number of points earned by the soccer team winning the Premier League in the season ending that May

How well does your model estimated from the first 5 years predict the Super Bowl score differential in last 5 years of your data? What is your conclusion?

Use a multiple regression model with the high temperatures on election day in these ten cities as explanatory variables for predicting the Democratic Party candidate's percentage of the total two-party vote for President in the 11 presidential elections 1980-2020: Claremont, California; Bozeman, Montana; Broken Bow, Nebraska; Burlington, Vermont; Caribou, Maine; Cody, Wyoming; Dover, Delaware; Elkins, West Virginia; Fargo, North Dakota; and Pocatello, Idaho. Now use your estimated model to predict the Democratic Party candidate's percentage of the total two-party vote for President in 2024. What is your conclusion?