Two-Sample Tests and Confidence Intervals

An A/B test is like an internet laboratory experiment. Two versions of a web page are created-typically the current page and a proposed modification, perhaps with a different banner, headline, or testimonial. A random number generator determines which page a user sees. A prespecified metric, like purchases or mailing-list signups, is used to determine the winning page.

The Obama 2008 presidential campaign used an A/B test to compare the number of web-page visitors who signed up for more information with these two buttons: "Sign Up" and "Learn More."

The results:

Under its original guidelines, the Environmental Protection Agency (EPA) concluded that second-hand smoke poses a cancer risk if a $95 \%$ confidence interval for the difference in the incidence of cancer in the control and treatment groups excluded zero. In 1996 the EPA changed this criterion to a $90 \%$ confidence interval. If, in fact, second-hand smoke does not pose a cancer risk, does a switch from a $95 \%$ to $90 \%$ confidence interval make it more or less likely that the EPA would conclude that second-hand smoke is a cancer risk?

A French hospital put 412 patients who had suffered one heart attack on a traditional Mediterranean diet (including olive oil, fruit, and bread); the control group consisted of 358 heart-attack patients who were given a recommended low-fat diet. Four of the patients on the Mediterranean diet and 17 of the patients on low-fat diet suffered a second heart attack during the 2 years of the study. Is the observed difference substantial and statistically persuasive?

A researcher compared the daily returns from a portfolio consisting of stocks that had been deleted from the Dow Jones Industrial Average with a portfolio consisting of the stocks that replaced them. Use these daily return data for a difference-in-means test and a matched-pair test. Which has the lower p-value? Why?

Evidence from many of sports indicate that performance is enhanced by our personal assessment of our ability. For example, when both contestants in an arm-wrestling study incorrectly believed the weaker person was stronger than his opponent, the weaker person won 10 of 12 matches. When the contestants had correct information about who was stronger, the stronger person won all 12 matches. Is this difference substantial and statistically persuasive?

A student played 100 games of Roshambo (rock, scissors, paper) against a computer software program, winning 32 games, losing 38, and drawing 30. The computer tried to figure out the player's strategy. This table separates the results into the first 50 games and the last 50 games:

A survey of 26 Division III athletes looked at their GPA during the semester they played their varsity sport and during the non-sport semester:

A difference-in-means test gave $\mathrm{t}=1.32$, with a two-sided p -value of .1939. A matched-pair test gave $\mathrm{t}=2.77$, with a two-sided p-value of .0104. Explain why these two tests can lead to different conclusions. Which test is more valid?

In a computer game, Blake is shown (in random order) either: (a) a rectangle that is yellow, blue, green, or red; or (b) the word yellow, blue, green, or red written in black. Blake's objective is to click as quickly as possible on the word at the bottom of the screen that matches the color or word shown in the middle of the screen. Here are the reaction times in seconds. Is the observed difference in average reaction time substantial and statistically persuasive?

The Unified Parkinson Disease Rating Scale (UPDRS) gauges the degree of Parkinson's disease in patients on a scale of $0-199$, with 0 representing no disability and 199 total disability. Eleven subjects with Parkinson's disease were put on a regiment of riding stationary bikes 3 days a week for 12 weeks. Are the results substantial and statistically persuasive?

Jerry's time in the 100 freestyle is normally distributed with a mean of 56 seconds and a standard deviation of 2 seconds; Kerry's time is normally distributed with a mean of 57 seconds and a standard deviation of 2.5 seconds. If their swim times are independent, what is the probability that Kerry will beat Jerry?

A study of 4 years of accident reports by the Dallas Fire Department found that fire trucks painted red made 153,348 runs and had 20 accidents, while fire trucks painted yellow made 135,035 runs and had 4 accidents. Assuming these to be independent random samples, is this difference substantial and statistically persuasive?

Explain the error(s) in this remark:

We used a matched-pair test instead of a difference-in-means test because a matched-pair test does not assume that the two samples have the same standard deviation.

In 1983 22,000 doctors participated in an experiment testing the effects of aspirin on the incidence of heart attacks. Half of the doctors were given a single aspirin tablet every other day; the other half took a placebo every other day. Neither the volunteers or the scientists conducting the experiment knew which volunteers were taking aspirin. After 5 years of what was intended to be a 7 -year test, the experiments were stopped by an outside committee monitoring the results. During this period, the doctors taking placebos had 18 fatal heart attacks and the doctors taking aspirin had 5. Is this difference substantial and statistically persuasive? Why do you suppose the experiment was stopped 2 years early?

In horseshoes, each player pitches two shoes in each inning. If the player pitches two ringers, this is called a double. A "hot hands" study of the 2000 World Horseshoe Pitching championships looked at whether players were more likely or less likely to throw a double after throwing a double in the previous inning. For example, female pitchers threw 1641 doubles and 1691 nondoubles after a nondouble inning, and threw 2142 doubles and 1666 nondoubles after a doubles inning. Is the difference substantial and statistically persuasive?

A treatment group was given a cold vaccine, while the control group received a placebo. Doctors then recorded the fraction of each group that caught a cold and calculated the two-sided p-value to be .08. Explain why you either agree or disagree with each of these interpretations of these results:

What is obviously wrong with this report of the results of a differ-ence-in-means test: "With 38 degrees of freedom, the t -value is 8.98 and the 2 -sided p -value is .2916. In this test, the null hypothesis cannot be rejected at the $1 \%$ or $5 \%$ level because the p -value is far too high."

Suppose that you run an experiment involving a treatment group and a control group with 20 subjects in each sample and a difference-inmeans test gives a $t$-value of 2.7 and a two-sided p-value of .01. Which of these statements are true and which are false?

A 2021 study compared the calories in 20 McDonald's menu items sold in the United States and the United Kingdom:

Students in one California school district are classified as either a gifted and talented education (GATE) student or a regular student. A survey of seventh- and eighth-graders found that 10 of 86 GATE students had mothers who did not attend college and that 35 of 124 regular students had mothers who did not attend college. Is this difference substantial and statistically persuasive?

College students rated their intelligence and preparedness for life after college on a scale of 1-10.

Explain why the proposed test is wrong and identify the correct test:

My question is "Does playing on home ice in the National Hockey League (NHL) give the home team a better chance to win?" I will choose six NHL teams and, for each team, use a Z single-sample categorical test of the null hypothesis that the chance of winning at home is equal to one half.

A researcher compared the Internet prices of 20 Abercrombie Fitch (AF) male clothing items with the prices of virtually identical clothes sold by Hollister, which is owned by AF. The average price of the 20 AF items was $\$ 50.13$, while the average price of the 20 Hollister items was $\$ 31.33$. The t -value was 3.38 using a two-sample test and 7.01 using a matched-pair test.

A researcher looked at the 20 English soccer clubs that played in the Premier League during the 2009-2010 season. These teams won an average of 9.65 out of 19 home games, with a standard deviation of 3.79, and won 4.55 out of 19 away games, with a standard deviation of 3.15. (Games that were not won were either lost or tied.) He then calculated the t -value for a two-sample test of the null hypothesis that the average number of wins in home games is equal to the average number of wins in away games:

California birth records were used to obtain data on the ZIP code residence of mothers when they gave birth to a daughter and the ZIP code residence of these daughters when they gave birth to a child. Of the 4,298 daughters of foreign-born mothers who lived in different ZIP codes, 2,579 lived in ZIP codes with higher median income than their mother's ZIP code and 1,719 lived in ZIP codes with lower income. For the 5,935 daughters of California-born mothers who lived in different ZIP codes, 3,181 lived in ZIP codes with higher median income than their mother's ZIP code and 2,754 lived in ZIP codes with lower median income. Is this difference between the daughters of foreign-born and California-born mothers substantial and statistically persuasive?

The 794 students in Pomona College's freshmen classes of 1987-1988 and 1988-1989 were separated into two groups: 552 who had attended public high schools and 242 who had attended private schools. Of those from public schools, $48(8.7 \% )$ were Pomona Scholars (an A-grade point average); of those from private schools, $13(5.4 \% )$ were Pomona Scholars. Does the observed difference seem substantial? The author calculated the Z-value to be 1.64 and concluded that although "two classes proved the null hypothesis to be true, ...the sample size is minuscule." Did she prove the null hypothesis to be true? Are her results invalidated by the small sample size?

About one-third of the applicants admitted to one small college actually enroll. Is there a difference between the average SAT scores of those who enroll and those who do not? Here are some recent data:

In the 1980s Kahneman and Tversky reported that when 200 people were asked the following question, 98 answered yes and 108 answered no:

Imagine that you have decided to see a play and paid the admission price of $\$ 10$ per ticket. As you enter the theater, you discover that you have lost the ticket. The seat is not marked and the ticket cannot be recovered. Would you pay $10 for another ticket? When 183 people were asked the following question, 161 answered yes and 22 answered no:

Imagine that you have decided to see a play where admission is $10 per ticket. As you enter the theater, you discover that you have lost a $10 bill. Would you still pay $10 for a ticket for the play? Is the difference in responses substantial and statistically persuasive?

Two polling organizations report these voter preferences:

Can the difference in these poll results be explained by sampling error? If you have no reason to believe one poll is more reliable than the other, what would you use for a prediction of each candidate's vote percentage?

One hundred randomly selected students were asked to rate the qualifications of two job candidates based on a resume. Half were given the resume for a mythical John Benson and half were given another resume, identical to the first in all respects except that the name was Mary Benson. The results:

Is the observed difference substantial and statistically persuasive?

Basketball players are often told that a missed shot usually bounces to the opposite side of the basket from which the shot is taken; for example, that a missed shot from the right side of the basket usually bounces to the left of the basket. A college player suspected that this tendency may be reversed at the end of a game, when the players are tired and their missed shots bounce off the front of the rim. A study of the first and last 10 minutes of videotapes of four different Division III male basketball teams yielded the data shown below. Use these data to see if the difference between the first and last 10 minutes is substantial and statistically persuasive.

Before coming to college, a student watched videotapes of a seminar, "Where there's a will, there's an A," which purports to show scientifically supported strategies for guaranteeing A grades. One strategy is to sit in the front of the class. To test this theory, a statistics student asked 50 randomly selected college students to write down their grade point average (GPA) and to indicate where they typically sit in large classrooms.

Data were obtained for the finals of the women's 800 -meter run at the Southern California Intercollegiate Athletic Conference Track and Field Championships. In three of the years studied (1983, 1986, and 1990), the meet was held on a dirt track; in three other years (1987, 1988, and 1989), an artificial surface was used. Is the observed difference in the average times substantial and statistically persuasive?

A study comparing the grades given by female and male students to an essay written by a male found that female graders gave an average grade of 7.125 and male graders gave an average grade of 7.500.

A random sample of 25 male college students and 25 female college students were each asked to name their favorite singer; 21 of the males named a male singer and 16 of the females named a male singer. Explain the error in this test of the null hypothesis that male and female students are equally likely to name a male singer: "The pooled proportion is $(21+16) /(25+25)=0.74$. We tested the null hypothesis $\mathrm{p}=0.74$ with the following Z statistic:"

From the 30 stocks in the Dow Jones Industrial Average, a researcher identified the ten stocks with the highest dividend-price (D/P) ratios and the ten stocks with the lowest D/P ratios on December 31, 2021. Here are the percentage returns over the next year:

A 2002 study compared radical prostatectomy to watchful waiting for the treatment of localized prostate cancer: 53 of 347 patients in the prostatectomy group died, as did 62 of 348 patients in the control group. The p -value for a test of the null hypothesis that there was no difference in the death rates was .367. Explain why you either agree or disagree with the conclusion of the US Preventive Services Task Force that there was "no difference between the two groups in overall mortality."