Lessons and exercises follow the school system and stay in its language.Les leçons et les exercices suivent le système scolaire et restent dans sa langue.Lektionen und Übungen folgen dem Schulsystem und bleiben in seiner Sprache.

Significance tests

Print the chapter

Every lesson of the chapter, ready for paper. Exercise sets are lettered A, B, C… and answered at the end.

To save a PDF, choose “Save as PDF” in the print dialog.

1. Stating hypotheses

Warm-up

Three quick questions from earlier lessons.

  1. In a random sample of 100, p̂ = 3/10. Build a 95% interval with z* = 1.96.
    1. The standard error:   ?\;?
    2. The margin of error:   ?\;?
    3. The lower end of the interval:   ?\;?
    4. The point estimate (center of the interval):
    5. With a sample 4 times as large, the standard error is divided by:
  2. Many random samples of size 161 are taken from the same population. Which quantity changes from sample to sample?
    1. The sample proportion p̂ (a statistic)
    2. The population proportion p (a parameter)
    3. The sample size
  3. Compared with samples of size 73, samples of size 292 give sample means that…
    1. vary less around the population mean
    2. vary more
    3. are more biased

Discover

A courtroom presumes innocence until the evidence is convincing. In a significance test, which hypothesis plays the role of "innocent"?

  1. The null hypothesis H₀
  2. The alternative Hₐ

Key idea

State the question as two hypotheses about a parameter.

H₀ (null): the parameter equals a specific value (no effect). Hₐ (alternative): what we look for evidence of: <<, >> (one-sided) or ≠\ne (two-sided). Hypotheses are about parameters (pp, μ\mu), never statistics.

Worked example

A factory says bulbs last 1,000 hours on average; a buyer suspects less.

H0:μ=1000H_0: \mu = 1000
Status quo.
Ha:μ<1000H_a: \mu < 1000
The suspicion.
One-sided test
Direction given.

Your turn, guided

Your turn: in "H₀: p = 0.3, Hₐ: p ≠ 0.3", what is the null value p0p_0?

p0=0.3p_0 = 0.3
The value in H₀.

Practice ★

  1. A company claims that 80% of customers are satisfied. A researcher suspects fewer are. Which is the null hypothesis?
    1. H₀: p = 0.80
    2. H₀: p < 0.80
    3. H₀: p̂ = 0.80
  1. The mean wait at a clinic used to be 81 minutes. A manager wants to know whether it has changed, in either direction. Which alternative?
    1. Hₐ: μ ≠ 81
    2. Hₐ: μ > 81
    3. Hₐ: x̄ ≠ 81

Watch out!

Practice ★★

  1. Why are hypotheses written about parameters such as p, not statistics such as p̂ (sample of 445)?
    1. Because we know p̂ exactly; the question is about the unknown population value
    2. Because statistics are harder to compute
    3. There is no difference between p and p̂

Exit ticket

Three questions to check you have got it.

  1. A company claims that 80% of customers are satisfied. A researcher suspects fewer are. Which is the null hypothesis?
    1. H₀: p = 0.80
    2. H₀: p < 0.80
    3. H₀: p̂ = 0.80
  2. A company claims that 60% of customers are satisfied. A researcher suspects fewer are. Which is the null hypothesis?
    1. H₀: p = 0.60
    2. H₀: p < 0.60
    3. H₀: p̂ = 0.60
  3. A company claims that 62% of customers are satisfied. A researcher suspects fewer are. Which is the null hypothesis?
    1. H₀: p = 0.62
    2. H₀: p < 0.62
    3. H₀: p̂ = 0.62

Summary card

Hypotheses

H₀: parameter = value. Hₐ: <, >, or ≠.

Watch out: parameters, never statistics.

One more step is optional: a challenge. Try it, or finish the lesson now.

Challenge ★★★

  1. H₀: p = 0.5. Assuming H₀ is true, how many successes do you expect in 200 trials?

2. The one-proportion z test and P-values

Warm-up

Three quick questions from earlier lessons.

  1. A company claims that 80% of customers are satisfied. A researcher suspects fewer are. Which is the null hypothesis?
    1. H₀: p = 0.80
    2. H₀: p < 0.80
    3. H₀: p̂ = 0.80
  2. The mean wait at a clinic used to be 81 minutes. A manager wants to know whether it has changed, in either direction. Which alternative?
    1. Hₐ: μ ≠ 81
    2. Hₐ: μ > 81
    3. Hₐ: x̄ ≠ 81
  3. Why are hypotheses written about parameters such as p, not statistics such as p̂ (sample of 445)?
    1. Because we know p̂ exactly; the question is about the unknown population value
    2. Because statistics are harder to compute
    3. There is no difference between p and p̂

Discover

A coin lands heads 60 times in 100 tosses. If it were fair, p^\hat{p} would have SD 0.05. How many SDs above 0.5 is 0.6?

  1. 2
  2. 0.1

Key idea

How surprising is the data if H₀ is true?

One-proportion z test: z=p^−p0p0(1−p0)/nz = \dfrac{\hat{p} - p_0}{\sqrt{p_0(1 - p_0)/n}} (the SE uses the null value p0p_0). The P-value is the probability, assuming H₀ is true, of a result at least as extreme as the one observed, in the direction(s) of Hₐ.

Worked example

H₀: p=0.25p = 0.25, n=300n = 300, p^=0.30\hat{p} = 0.30.

SE=0.025SE = 0.025
√(0.25 × 0.75/300).
z=2z = 2
(0.30 − 0.25)/0.025.
P-value ≈0.023\approx 0.023
Upper tail for Hₐ: p > 0.25.

Your turn, guided

Your turn: SE = 0.04, p^−p0=0.06\hat{p} - p_0 = 0.06. Find zz.

z=1.5z = 1.5
0.06/0.04.

Practice ★

  1. Test H₀: p = 0.5 with a random sample of 100 in which p̂ = 52/100. Find the z statistic.
  2. Test H₀: p = 0.5 with a random sample of 100 in which p̂ = 58/100. Find the z statistic.
  1. Test H₀: p = 0.2 with n = 400 and p̂ = 28/100. Find the z statistic.

Watch out!

Practice ★★

  1. A test gives a P-value of 0.05. What does it mean?
    1. If H₀ were true, a result at least as extreme as ours would happen with probability 0.05
    2. There is a 0.05 probability that H₀ is true
    3. The result is due to chance with probability 0.05

Exit ticket

Three questions to check you have got it.

  1. Test H₀: p = 0.5 with a random sample of 100 in which p̂ = 52/100. Find the z statistic.
  2. Test H₀: p = 0.5 with a random sample of 100 in which p̂ = 58/100. Find the z statistic.
  3. Test H₀: p = 0.5 with a random sample of 100 in which p̂ = 56/100. Find the z statistic.

Summary card

z test for p

z=(p^−p0)/p0(1−p0)/nz = (\hat{p} - p_0)/\sqrt{p_0(1 - p_0)/n}; P-value assumes H₀.

Watch out: use p0p_0 in the SE.

One more step is optional: a challenge. Try it, or finish the lesson now.

Challenge ★★★

  1. Test H₀: p = 0.5 with a random sample of 2,500 in which p̂ = 60/100. Find the z statistic.

3. Conclusions, errors and power

Warm-up

Three quick questions from earlier lessons.

  1. Test H₀: p = 0.5 with a random sample of 100 in which p̂ = 52/100. Find the z statistic.
  2. Test H₀: p = 0.2 with n = 400 and p̂ = 28/100. Find the z statistic.
  3. A test gives a P-value of 0.05. What does it mean?
    1. If H₀ were true, a result at least as extreme as ours would happen with probability 0.05
    2. There is a 0.05 probability that H₀ is true
    3. The result is due to chance with probability 0.05

Discover

At α = 0.05 a test gives P-value 0.30. Does this prove H₀ is true?

  1. No: we just lack evidence against it
  2. Yes

Key idea

Decide, and know what can go wrong.

If P-value ≤α\le \alpha, reject H₀ (convincing evidence for Hₐ); otherwise fail to reject H₀. Type I error: rejecting a true H₀ (probability α). Type II error: failing to reject a false H₀. Power = probability of correctly rejecting a false H₀; it grows with nn and with α.

Worked example

H₀: a new teaching method is no better.

Type I: adopting a method that is not better
Reject a true H₀.
Type II: keeping the old method when the new one is better
Miss a real improvement.
Higher power, fewer Type II errors
More students in the study.

Your turn, guided

Your turn: α = 0.01. What is the probability of a Type I error when H₀ is true?

0.010.01
It equals α.

Practice ★

  1. At significance level α = 0.05, a test gives P-value 0.04. What do you conclude?
    1. Reject H₀: there is convincing evidence for Hₐ
    2. Accept H₀
    3. Hₐ is proven true
  1. A test of a new drug (sample of 388) has H₀: the drug has no effect. What is a Type I error?
    1. Concluding the drug works when it really has no effect
    2. Concluding there is no effect when the drug really works
    3. Making a mistake in the data

Watch out!

Practice ★★

  1. How can a researcher increase the power of a test that now uses 82 subjects?
    1. Use a larger sample
    2. Use a smaller α
    3. Use fewer subjects

Exit ticket

Three questions to check you have got it.

  1. At significance level α = 0.05, a test gives P-value 0.04. What do you conclude?
    1. Reject H₀: there is convincing evidence for Hₐ
    2. Accept H₀
    3. Hₐ is proven true
  2. At significance level α = 0.05, a test gives P-value 0.03. What do you conclude?
    1. Reject H₀: there is convincing evidence for Hₐ
    2. Accept H₀
    3. Hₐ is proven true
  3. At significance level α = 0.05, a test gives P-value 0.02. What do you conclude?
    1. Reject H₀: there is convincing evidence for Hₐ
    2. Accept H₀
    3. Hₐ is proven true

Summary card

Conclusions and errors

  • P ≤ α: reject H₀. Otherwise: fail to reject.
  • Type I: reject true H₀. Type II: keep false H₀.
  • Power rises with nn.

Watch out: never "accept" or "prove" H₀.

One more step is optional: a challenge. Try it, or finish the lesson now.

Challenge ★★★

  1. A test uses significance level α = 0.07. If H₀ is true, what is the probability of a Type I error?

4. The one-sample t test for a mean

Warm-up

Three quick questions from earlier lessons.

  1. A random sample of 25 has standard deviation s = 16. With t* = 2.064 (24 df, 95%), find the margin of error for the mean.
  2. A one-sample t interval uses a sample of 10. How many degrees of freedom?
  3. Why is a t critical value used for a confidence interval for a mean from a sample of 40?
    1. Because σ is estimated by s, which adds variability
    2. Because n is less than 1,000
    3. Because the data are skewed

Discover

H₀: μ = 50. A sample of 16 has x̄ = 53 and s = 8. How many standard errors is 53 above 50?

  1. 1.5
  2. 0.375

Key idea

The one-sample t test.

t=xˉ−μ0s/nt = \dfrac{\bar{x} - \mu_0}{s/\sqrt{n}} with n−1n - 1 degrees of freedom. For a two-sided Hₐ, the P-value counts both tails (double the one-tail area).

Worked example

H₀: μ = 100, Hₐ: μ ≠ 100; n=25n = 25, xˉ=104\bar{x} = 104, s=10s = 10.

SE=2SE = 2
10/5.
t=2t = 2, df =24= 24
4/2.
P-value ≈2×0.028=0.057\approx 2 \times 0.028 = 0.057
Both tails.

Your turn, guided

Your turn: one-tail area 0.04, two-sided test. The P-value.

0.080.08
2 × 0.04.

Practice ★

  1. Test H₀: μ = 39. A random sample of 9 has x̄ = 43 and s = 9. Find the t statistic.
  2. Test H₀: μ = 24. A random sample of 64 has x̄ = 29 and s = 18. Find the t statistic.
  1. For a test statistic, the area in one tail beyond it is 0.04. Find the P-value of the two-sided test.

Watch out!

Practice ★★

  1. A t test gives P-value 0.20. What can you conclude at α = 0.05?
    1. Fail to reject H₀: not enough evidence for Hₐ
    2. H₀ is true
    3. Reject H₀

Exit ticket

Three questions to check you have got it.

  1. Test H₀: μ = 39. A random sample of 9 has x̄ = 43 and s = 9. Find the t statistic.
  2. Test H₀: μ = 24. A random sample of 64 has x̄ = 29 and s = 18. Find the t statistic.
  3. Test H₀: μ = 71. A random sample of 9 has x̄ = 76 and s = 8. Find the t statistic.

Summary card

t test for μ

t=(xˉ−μ0)/(s/n)t = (\bar{x} - \mu_0)/(s/\sqrt{n}), df =n−1= n - 1; two-sided: double the tail.

Watch out: failing to reject is not proving H₀.

One more step is optional: a challenge. Try it, or finish the lesson now.

Challenge ★★★

  1. Test H₀: μ = 33. A random sample of 25 has x̄ = 36 and s = 19. Find the t statistic.

5. Comparing two groups: two proportions and paired data

Warm-up

Three quick questions from earlier lessons.

  1. Test H₀: p = 0.5 with a random sample of 100 in which p̂ = 52/100. Find the z statistic.
  2. Test H₀: p = 0.2 with n = 400 and p̂ = 28/100. Find the z statistic.
  3. A test gives a P-value of 0.05. What does it mean?
    1. If H₀ were true, a result at least as extreme as ours would happen with probability 0.05
    2. There is a 0.05 probability that H₀ is true
    3. The result is due to chance with probability 0.05

Discover

To test whether two groups have the same success rate, group A had 30 of 100 and group B 50 of 100. Under H₀ (same rate), what is the best estimate of that common rate?

  1. 0.4
  2. It cannot be estimated

Key idea

Two independent groups, or one group measured twice.

Two proportions: pooled p^=x1+x2n1+n2\hat{p} = \dfrac{x_1 + x_2}{n_1 + n_2}, z=p^1−p^2p^(1−p^)(1/n1+1/n2)z = \dfrac{\hat{p}_1 - \hat{p}_2}{\sqrt{\hat{p}(1 - \hat{p})(1/n_1 + 1/n_2)}}. Paired data (same subjects twice): a one-sample t test on the differences.

Worked example

x1=30x_1 = 30, n1=100n_1 = 100; x2=50x_2 = 50, n2=100n_2 = 100.

p^=0.4\hat{p} = 0.4
Pool.
SE≈0.069SE \approx 0.069
√(0.4 × 0.6 × 0.02).
z≈−2.89z \approx -2.89
(0.3 − 0.5)/0.069.

Your turn, guided

Your turn: 20 of 50 and 30 of 50. The pooled proportion.

0.50.5
50/100.

Practice ★

  1. To test H₀: p₁ = p₂, group 1 had 21 successes out of 82 and group 2 had 22 out of 102. Find the pooled proportion.
  2. To test H₀: p₁ = p₂, group 1 had 51 successes out of 85 and group 2 had 51 out of 105. Find the pooled proportion.
  1. 24 students take a test before and after a course. Which procedure compares the scores?
    1. A paired t test on the 24 differences (after − before)
    2. A two-sample t test treating the groups as independent
    3. A two-proportion z test

Watch out!

Practice ★★

  1. Testing H₀: p₁ = p₂ with n₁ = n₂ = 100, the pooled proportion is 3/10. Find the standard error of p̂₁ − p̂₂.

Exit ticket

Three questions to check you have got it.

  1. To test H₀: p₁ = p₂, group 1 had 21 successes out of 82 and group 2 had 22 out of 102. Find the pooled proportion.
  2. To test H₀: p₁ = p₂, group 1 had 51 successes out of 85 and group 2 had 51 out of 105. Find the pooled proportion.
  3. To test H₀: p₁ = p₂, group 1 had 32 successes out of 93 and group 2 had 53 out of 113. Find the pooled proportion.

Summary card

Comparing two groups

  • Two proportions: pooled p^\hat{p} in the SE.
  • Paired: test the differences.

Watch out: pool counts, not proportions.

One more step is optional: a challenge. Try it, or finish the lesson now.

Challenge ★★★

  1. Group 1 had 41 successes out of 100 and group 2 had 26 out of 100. Find p^1−p^2\hat{p}_1 - \hat{p}_2.

6. Chi-square tests

Warm-up

Three quick questions from earlier lessons.

  1. A coin is tossed 100 times and lands heads 63 times. Test H₀: p = 0.5 against Hₐ: p > 0.5.
    1. The standard error under H₀:   ?\;?
    2. The z statistic:   ?\;?
    3. The proportion of heads needed for z=2z = 2:   ?\;?
    4. The value of p stated by the null hypothesis:
    5. At α = 0.05 (one-sided, z* = 1.645), what is the smallest number of heads that leads to rejecting H₀?
  2. At significance level α = 0.05, a test gives P-value 0.04. What do you conclude?
    1. Reject H₀: there is convincing evidence for Hₐ
    2. Accept H₀
    3. Hₐ is proven true
  3. A test of a new drug (sample of 388) has H₀: the drug has no effect. What is a Type I error?
    1. Concluding the drug works when it really has no effect
    2. Concluding there is no effect when the drug really works
    3. Making a mistake in the data

Discover

A die is rolled 60 times. If it is fair, how many times do you expect each face?

  1. 10
  2. 60

Key idea

Compare observed counts with expected counts.

χ2=∑(O−E)2E\chi^2 = \sum \dfrac{(O - E)^2}{E}. Goodness of fit: E=n×piE = n \times p_i, df == (categories −1- 1). Two-way table (independence or homogeneity): E=row total×column totalgrand totalE = \dfrac{\text{row total} \times \text{column total}}{\text{grand total}}, df =(r−1)(c−1)= (r - 1)(c - 1). Large χ2\chi^2 (small P-value) is evidence against H₀. Condition: all expected counts at least 5.

Worked example

Die: observed 14 sixes, expected 10.

Contribution =1.6= 1.6
(14 − 10)²/10.
df =5= 5
Six faces.
χ2\chi^2 is the sum
Add all six contributions.

Your turn, guided

Your turn: a 3 × 4 two-way table. Degrees of freedom.

66
(3 − 1)(4 − 1).

Practice ★

  1. A die is rolled 60 times to test whether it is fair. What is the expected count for each face?
  1. In a chi-square test, one category has observed count 39 and expected count 21. Find its contribution to χ².
  2. In a chi-square test, one category has observed count 16 and expected count 27. Find its contribution to χ².

Watch out!

Practice ★★

  1. In a two-way table with grand total 158, a row total is 55 and a column total is 41. Find the expected count in that cell under independence.
  2. In a two-way table with grand total 134, a row total is 53 and a column total is 40. Find the expected count in that cell under independence.

Exit ticket

Three questions to check you have got it.

  1. In a chi-square test, one category has observed count 39 and expected count 21. Find its contribution to χ².
  2. In a chi-square test, one category has observed count 16 and expected count 27. Find its contribution to χ².
  3. In a chi-square test, one category has observed count 20 and expected count 16. Find its contribution to χ².

Summary card

Chi-square tests

χ2=∑(O−E)2/E\chi^2 = \sum (O - E)^2/E; GOF: E=npE = np; two-way: E=E = row × column / total.

Watch out: divide by EE; use counts, not percentages.

One more step is optional: a challenge. Try it, or finish the lesson now.

Challenge ★★★

  1. A chi-square test of independence uses a table with 5 rows and 5 columns. How many degrees of freedom does it have?

7. Exact p-values for a coin

Discover

You suspect a coin favors heads. You flip it 8 times and get 8 heads. If the coin were fair, how likely is that?

  1. (12)8=1256≈0.4 %\left(\tfrac{1}{2}\right)^8 = \tfrac{1}{256} \approx 0.4\,\%
  2. 816=50 %\tfrac{8}{16} = 50\,\%

Key idea

Exact p-values for a coin.

For nn flips of a fair coin:

  • P(all n heads)=12nP(\text{all } n \text{ heads}) = \dfrac{1}{2^n};
  • P(at least n−1 heads)=n+12nP(\text{at least } n - 1 \text{ heads}) = \dfrac{n + 1}{2^n}.

These are one-sided p-values for 'the coin favors heads'.

Worked example

10 flips, 9 or more heads.

1+10=111 + 10 = 11 out of 210=10242^{10} = 1024
Count the outcomes.
≈0.011\approx 0.011: significant at the 5 % level
p-value.

Your turn, guided

Your turn: 5 flips, all heads: p-value as a fraction?

132\dfrac{1}{32}
125\dfrac{1}{2^5}.

Practice ★

  1. A fair coin is flipped 11 times. Find the probability (as a fraction) that all 11 flips are heads.
  2. A fair coin is flipped 7 times. Find the probability (as a fraction) that all 7 flips are heads.
  3. A fair coin is flipped 10 times. Find the probability (as a fraction) that all 10 flips are heads.

Practice ★★

  1. A fair coin is flipped 5 times. Find the p-value (as a fraction) for getting at least 4 heads.
  2. A fair coin is flipped 7 times. Find the p-value (as a fraction) for getting at least 6 heads.

Exit ticket

Two questions to check.

  1. A fair coin is flipped 11 times. Find the probability (as a fraction) that all 11 flips are heads.
  2. A fair coin is flipped 7 times. Find the probability (as a fraction) that all 7 flips are heads.

Summary card

Coin p-values

  • All heads: 12n\dfrac{1}{2^n}.
  • At least n−1n - 1: n+12n\dfrac{n + 1}{2^n}.

Sign tests in medicine use exactly this idea: did more patients improve than chance allows?

8. History of significance tests: Arbuthnot, Pearson and Fisher

Discover

Records from 1629 to 1710 showed slightly more boys than girls born. In 1710 John Arbuthnot asked: could chance alone explain that? Suppose boys outnumbered girls in every one of those years.

  1. Hardly: that would be like 82 heads in a row
  2. Yes, easily

Key idea

Testing hypotheses, from baptisms to fields.

1700175018001850190019501710Arbuthnot1710Arbuthnot
  1. Step 11700175018001850190019501710Arbuthnot1710ArbuthnotJohn Arbuthnot 1667–1735Godfrey Kneller, 1723. Public domain1710: Arbuthnot's paper on male and female births, a first test of significance.
  2. Step 21700175018001850190019501710Arbuthnot1900Pearson1710Arbuthnot1900PearsonKarl Pearson 1857–1936Unknown artist, 1910. Public domain1900: Pearson's chi-square test.
  3. Step 31700175018001850190019501710Arbuthnot1900Pearson1935Fisher1710Arbuthnot1900Pearson1935FisherRonald Fisher 1890–1962Barry Eagel, 1952. Barry Eagel. CC BY-SA 4.01935: Fisher's The Design of Experiments.

In 1710 John Arbuthnot (1667–1735) studied the slight excess of male births over female births from 1629 to 1710, perhaps the first application of probability to social statistics and the first formal test of significance. In 1900 Karl Pearson published the chi-square test. At Rothamsted, Ronald Fisher developed the design and analysis of experiments.

Worked example

Arbuthnot's reasoning in numbers: how many years from 1629 to 1710, inclusive?

1710−1629+1=821710 - 1629 + 1 = 82
Suppose more boys every year: count the years.
(12)82≈2×10−25\left(\tfrac{1}{2}\right)^{82} \approx 2 \times 10^{-25}
Probability by chance alone.

Practice ★

  1. What did Arbuthnot study in his 1710 paper?
    1. The excess of male over female births
    2. Fertilizers on randomized plots
    3. The chi-square test
  1. Like Arbuthnot: if each year were a fair 'coin flip' between more boys and more girls, what is the probability (as a fraction) of more boys 5 years in a row?
  2. Like Arbuthnot: if each year were a fair 'coin flip' between more boys and more girls, what is the probability (as a fraction) of more boys 10 years in a row?
  1. Pearson's chi-square: one category has expected count 10 and observed count 12. Compute its contribution (O−E)2E\dfrac{(O - E)^2}{E}.

Summary card

Significance tests today

  • Medicine: does a new drug work better than a placebo?
  • Websites: does a new design get more clicks (A/B tests)?
  • Quality control: is a machine still making parts to specification?

Sources: MacTutor biographies of Arbuthnot, Pearson and Fisher (see "Sources and credits").