Lessons and exercises follow the school system and stay in its language.Les leçons et les exercices suivent le système scolaire et restent dans sa langue.Lektionen und Übungen folgen dem Schulsystem und bleiben in seiner Sprache.
Every lesson of the topic, from the first year to the last, ready for paper. Exercise sets are lettered A, B, C… and answered at the end.
To save a PDF, choose “Save as PDF” in the print dialog.
Statistics and probability
Significance tests
NicknameDate
Scan to open onlineenglish.sedifier.com/en-us/branch/s/sig/print/
1. Stating hypotheses
Warm-up
Three quick questions from earlier lessons.
In a random sample of 100, p̂ = 3/10. Build a 95% interval with z* = 1.96.
The standard error: ?
The margin of error: ?
The lower end of the interval: ?
The point estimate (center of the interval):
With a sample 4 times as large, the standard error is divided by:
Many random samples of size 161 are taken from the same population. Which quantity changes from sample to sample?
The sample proportion p̂ (a statistic)
The population proportion p (a parameter)
The sample size
Compared with samples of size 73, samples of size 292 give sample means that…
vary less around the population mean
vary more
are more biased
Discover
A courtroom presumes innocence until the evidence is convincing. In a significance test, which hypothesis plays the role of "innocent"?
The null hypothesis H₀
The alternative Hₐ
Key idea
State the question as two hypotheses about a parameter.
H₀ (null): the parameter equals a specific value (no effect). Hₐ (alternative): what we look for evidence of: <, > (one-sided) or = (two-sided). Hypotheses are about parameters (p, μ), never statistics.
Worked example
A factory says bulbs last 1,000 hours on average; a buyer suspects less.
H0:μ=1000
Status quo.
Ha:μ<1000
The suspicion.
One-sided test
Direction given.
Your turn, guided
Your turn: in "H₀: p = 0.3, Hₐ: p ≠ 0.3", what is the null value p0?
p0=0.3
The value in H₀.
Practice ★
A company claims that 80% of customers are satisfied. A researcher suspects fewer are. Which is the null hypothesis?
H₀: p = 0.80
H₀: p < 0.80
H₀: p̂ = 0.80
The mean wait at a clinic used to be 81 minutes. A manager wants to know whether it has changed, in either direction. Which alternative?
Hₐ: μ ≠ 81
Hₐ: μ > 81
Hₐ: x̄ ≠ 81
Watch out!
Practice ★★
Why are hypotheses written about parameters such as p, not statistics such as p̂ (sample of 445)?
Because we know p̂ exactly; the question is about the unknown population value
Because statistics are harder to compute
There is no difference between p and p̂
Exit ticket
Three questions to check you have got it.
A company claims that 80% of customers are satisfied. A researcher suspects fewer are. Which is the null hypothesis?
H₀: p = 0.80
H₀: p < 0.80
H₀: p̂ = 0.80
A company claims that 60% of customers are satisfied. A researcher suspects fewer are. Which is the null hypothesis?
H₀: p = 0.60
H₀: p < 0.60
H₀: p̂ = 0.60
A company claims that 62% of customers are satisfied. A researcher suspects fewer are. Which is the null hypothesis?
H₀: p = 0.62
H₀: p < 0.62
H₀: p̂ = 0.62
Summary card
Hypotheses
H₀: parameter = value. Hₐ: <, >, or ≠.
Watch out: parameters, never statistics.
One more step is optional: a challenge. Try it, or finish the lesson now.
Challenge ★★★
H₀: p = 0.5. Assuming H₀ is true, how many successes do you expect in 200 trials?
2. The one-proportion z test and P-values
Warm-up
Three quick questions from earlier lessons.
A company claims that 80% of customers are satisfied. A researcher suspects fewer are. Which is the null hypothesis?
H₀: p = 0.80
H₀: p < 0.80
H₀: p̂ = 0.80
The mean wait at a clinic used to be 81 minutes. A manager wants to know whether it has changed, in either direction. Which alternative?
Hₐ: μ ≠ 81
Hₐ: μ > 81
Hₐ: x̄ ≠ 81
Why are hypotheses written about parameters such as p, not statistics such as p̂ (sample of 445)?
Because we know p̂ exactly; the question is about the unknown population value
Because statistics are harder to compute
There is no difference between p and p̂
Discover
A coin lands heads 60 times in 100 tosses. If it were fair, p^ would have SD 0.05. How many SDs above 0.5 is 0.6?
2
0.1
Key idea
How surprising is the data if H₀ is true?
One-proportion z test:z=p0(1−p0)/np^−p0 (the SE uses the null valuep0). The P-value is the probability, assuming H₀ is true, of a result at least as extreme as the one observed, in the direction(s) of Hₐ.
Worked example
H₀: p=0.25, n=300, p^=0.30.
SE=0.025
√(0.25 × 0.75/300).
z=2
(0.30 − 0.25)/0.025.
P-value ≈0.023
Upper tail for Hₐ: p > 0.25.
Your turn, guided
Your turn: SE = 0.04, p^−p0=0.06. Find z.
z=1.5
0.06/0.04.
Practice ★
Test H₀: p = 0.5 with a random sample of 100 in which p̂ = 52/100. Find the z statistic.
Test H₀: p = 0.5 with a random sample of 100 in which p̂ = 58/100. Find the z statistic.
Test H₀: p = 0.2 with n = 400 and p̂ = 28/100. Find the z statistic.
Watch out!
Practice ★★
A test gives a P-value of 0.05. What does it mean?
If H₀ were true, a result at least as extreme as ours would happen with probability 0.05
There is a 0.05 probability that H₀ is true
The result is due to chance with probability 0.05
Exit ticket
Three questions to check you have got it.
Test H₀: p = 0.5 with a random sample of 100 in which p̂ = 52/100. Find the z statistic.
Test H₀: p = 0.5 with a random sample of 100 in which p̂ = 58/100. Find the z statistic.
Test H₀: p = 0.5 with a random sample of 100 in which p̂ = 56/100. Find the z statistic.
Summary card
z test for p
z=(p^−p0)/p0(1−p0)/n; P-value assumes H₀.
Watch out: use p0 in the SE.
One more step is optional: a challenge. Try it, or finish the lesson now.
Challenge ★★★
Test H₀: p = 0.5 with a random sample of 2,500 in which p̂ = 60/100. Find the z statistic.
3. Comparing two groups: two proportions and paired data
Warm-up
Three quick questions from earlier lessons.
Test H₀: p = 0.5 with a random sample of 100 in which p̂ = 52/100. Find the z statistic.
Test H₀: p = 0.2 with n = 400 and p̂ = 28/100. Find the z statistic.
A test gives a P-value of 0.05. What does it mean?
If H₀ were true, a result at least as extreme as ours would happen with probability 0.05
There is a 0.05 probability that H₀ is true
The result is due to chance with probability 0.05
Discover
To test whether two groups have the same success rate, group A had 30 of 100 and group B 50 of 100. Under H₀ (same rate), what is the best estimate of that common rate?
0.4
It cannot be estimated
Key idea
Two independent groups, or one group measured twice.
Two proportions: pooled p^=n1+n2x1+x2, z=p^(1−p^)(1/n1+1/n2)p^1−p^2. Paired data (same subjects twice): a one-sample t test on the differences.
Worked example
x1=30, n1=100; x2=50, n2=100.
p^=0.4
Pool.
SE≈0.069
√(0.4 × 0.6 × 0.02).
z≈−2.89
(0.3 − 0.5)/0.069.
Your turn, guided
Your turn: 20 of 50 and 30 of 50. The pooled proportion.
0.5
50/100.
Practice ★
To test H₀: p₁ = p₂, group 1 had 21 successes out of 82 and group 2 had 22 out of 102. Find the pooled proportion.
To test H₀: p₁ = p₂, group 1 had 51 successes out of 85 and group 2 had 51 out of 105. Find the pooled proportion.
24 students take a test before and after a course. Which procedure compares the scores?
A paired t test on the 24 differences (after − before)
A two-sample t test treating the groups as independent
A two-proportion z test
Watch out!
Practice ★★
Testing H₀: p₁ = p₂ with n₁ = n₂ = 100, the pooled proportion is 3/10. Find the standard error of p̂₁ − p̂₂.
Exit ticket
Three questions to check you have got it.
To test H₀: p₁ = p₂, group 1 had 21 successes out of 82 and group 2 had 22 out of 102. Find the pooled proportion.
To test H₀: p₁ = p₂, group 1 had 51 successes out of 85 and group 2 had 51 out of 105. Find the pooled proportion.
To test H₀: p₁ = p₂, group 1 had 32 successes out of 93 and group 2 had 53 out of 113. Find the pooled proportion.
Summary card
Comparing two groups
Two proportions: pooled p^ in the SE.
Paired: test the differences.
Watch out: pool counts, not proportions.
One more step is optional: a challenge. Try it, or finish the lesson now.
Challenge ★★★
Group 1 had 41 successes out of 100 and group 2 had 26 out of 100. Find p^1−p^2.
4. Conclusions, errors and power
Warm-up
Three quick questions from earlier lessons.
Test H₀: p = 0.5 with a random sample of 100 in which p̂ = 52/100. Find the z statistic.
Test H₀: p = 0.2 with n = 400 and p̂ = 28/100. Find the z statistic.
A test gives a P-value of 0.05. What does it mean?
If H₀ were true, a result at least as extreme as ours would happen with probability 0.05
There is a 0.05 probability that H₀ is true
The result is due to chance with probability 0.05
Discover
At α = 0.05 a test gives P-value 0.30. Does this prove H₀ is true?
No: we just lack evidence against it
Yes
Key idea
Decide, and know what can go wrong.
If P-value ≤α, reject H₀ (convincing evidence for Hₐ); otherwise fail to reject H₀. Type I error: rejecting a true H₀ (probability α). Type II error: failing to reject a false H₀. Power = probability of correctly rejecting a false H₀; it grows with n and with α.
Worked example
H₀: a new teaching method is no better.
Type I: adopting a method that is not better
Reject a true H₀.
Type II: keeping the old method when the new one is better
Miss a real improvement.
Higher power, fewer Type II errors
More students in the study.
Your turn, guided
Your turn: α = 0.01. What is the probability of a Type I error when H₀ is true?
0.01
It equals α.
Practice ★
At significance level α = 0.05, a test gives P-value 0.04. What do you conclude?
Reject H₀: there is convincing evidence for Hₐ
Accept H₀
Hₐ is proven true
A test of a new drug (sample of 388) has H₀: the drug has no effect. What is a Type I error?
Concluding the drug works when it really has no effect
Concluding there is no effect when the drug really works
Making a mistake in the data
Watch out!
Practice ★★
How can a researcher increase the power of a test that now uses 82 subjects?
Use a larger sample
Use a smaller α
Use fewer subjects
Exit ticket
Three questions to check you have got it.
At significance level α = 0.05, a test gives P-value 0.04. What do you conclude?
Reject H₀: there is convincing evidence for Hₐ
Accept H₀
Hₐ is proven true
At significance level α = 0.05, a test gives P-value 0.03. What do you conclude?
Reject H₀: there is convincing evidence for Hₐ
Accept H₀
Hₐ is proven true
At significance level α = 0.05, a test gives P-value 0.02. What do you conclude?
Reject H₀: there is convincing evidence for Hₐ
Accept H₀
Hₐ is proven true
Summary card
Conclusions and errors
P ≤ α: reject H₀. Otherwise: fail to reject.
Type I: reject true H₀. Type II: keep false H₀.
Power rises with n.
Watch out: never "accept" or "prove" H₀.
One more step is optional: a challenge. Try it, or finish the lesson now.
Challenge ★★★
A test uses significance level α = 0.07. If H₀ is true, what is the probability of a Type I error?
5. The one-sample t test for a mean
Warm-up
Three quick questions from earlier lessons.
A random sample of 25 has standard deviation s = 16. With t* = 2.064 (24 df, 95%), find the margin of error for the mean.
A one-sample t interval uses a sample of 10. How many degrees of freedom?
Why is a t critical value used for a confidence interval for a mean from a sample of 40?
Because σ is estimated by s, which adds variability
Because n is less than 1,000
Because the data are skewed
Discover
H₀: μ = 50. A sample of 16 has x̄ = 53 and s = 8. How many standard errors is 53 above 50?
1.5
0.375
Key idea
The one-sample t test.
t=s/nxˉ−μ0 with n−1 degrees of freedom. For a two-sided Hₐ, the P-value counts both tails (double the one-tail area).
Worked example
H₀: μ = 100, Hₐ: μ ≠ 100; n=25, xˉ=104, s=10.
SE=2
10/5.
t=2, df =24
4/2.
P-value ≈2×0.028=0.057
Both tails.
Your turn, guided
Your turn: one-tail area 0.04, two-sided test. The P-value.
0.08
2 × 0.04.
Practice ★
Test H₀: μ = 39. A random sample of 9 has x̄ = 43 and s = 9. Find the t statistic.
Test H₀: μ = 24. A random sample of 64 has x̄ = 29 and s = 18. Find the t statistic.
For a test statistic, the area in one tail beyond it is 0.04. Find the P-value of the two-sided test.
Watch out!
Practice ★★
A t test gives P-value 0.20. What can you conclude at α = 0.05?
Fail to reject H₀: not enough evidence for Hₐ
H₀ is true
Reject H₀
Exit ticket
Three questions to check you have got it.
Test H₀: μ = 39. A random sample of 9 has x̄ = 43 and s = 9. Find the t statistic.
Test H₀: μ = 24. A random sample of 64 has x̄ = 29 and s = 18. Find the t statistic.
Test H₀: μ = 71. A random sample of 9 has x̄ = 76 and s = 8. Find the t statistic.
Summary card
t test for μ
t=(xˉ−μ0)/(s/n), df =n−1; two-sided: double the tail.
Watch out: failing to reject is not proving H₀.
One more step is optional: a challenge. Try it, or finish the lesson now.
Challenge ★★★
Test H₀: μ = 33. A random sample of 25 has x̄ = 36 and s = 19. Find the t statistic.
6. Chi-square tests
Warm-up
Three quick questions from earlier lessons.
A coin is tossed 100 times and lands heads 63 times. Test H₀: p = 0.5 against Hₐ: p > 0.5.
The standard error under H₀: ?
The z statistic: ?
The proportion of heads needed for z=2: ?
The value of p stated by the null hypothesis:
At α = 0.05 (one-sided, z* = 1.645), what is the smallest number of heads that leads to rejecting H₀?
At significance level α = 0.05, a test gives P-value 0.04. What do you conclude?
Reject H₀: there is convincing evidence for Hₐ
Accept H₀
Hₐ is proven true
A test of a new drug (sample of 388) has H₀: the drug has no effect. What is a Type I error?
Concluding the drug works when it really has no effect
Concluding there is no effect when the drug really works
Making a mistake in the data
Discover
A die is rolled 60 times. If it is fair, how many times do you expect each face?
10
60
Key idea
Compare observed counts with expected counts.
χ2=∑E(O−E)2. Goodness of fit:E=n×pi, df = (categories −1). Two-way table (independence or homogeneity): E=grand totalrow total×column total, df =(r−1)(c−1). Large χ2 (small P-value) is evidence against H₀. Condition: all expected counts at least 5.
Worked example
Die: observed 14 sixes, expected 10.
Contribution =1.6
(14 − 10)²/10.
df =5
Six faces.
χ2 is the sum
Add all six contributions.
Your turn, guided
Your turn: a 3 × 4 two-way table. Degrees of freedom.
6
(3 − 1)(4 − 1).
Practice ★
A die is rolled 60 times to test whether it is fair. What is the expected count for each face?
In a chi-square test, one category has observed count 39 and expected count 21. Find its contribution to χ².
In a chi-square test, one category has observed count 16 and expected count 27. Find its contribution to χ².
Watch out!
Practice ★★
In a two-way table with grand total 158, a row total is 55 and a column total is 41. Find the expected count in that cell under independence.
In a two-way table with grand total 134, a row total is 53 and a column total is 40. Find the expected count in that cell under independence.
Exit ticket
Three questions to check you have got it.
In a chi-square test, one category has observed count 39 and expected count 21. Find its contribution to χ².
In a chi-square test, one category has observed count 16 and expected count 27. Find its contribution to χ².
In a chi-square test, one category has observed count 20 and expected count 16. Find its contribution to χ².
Watch out: divide by E; use counts, not percentages.
One more step is optional: a challenge. Try it, or finish the lesson now.
Challenge ★★★
A chi-square test of independence uses a table with 5 rows and 5 columns. How many degrees of freedom does it have?
7. Exact p-values for a coin
Discover
You suspect a coin favors heads. You flip it 8 times and get 8 heads. If the coin were fair, how likely is that?
(21)8=2561≈0.4%
168=50%
Key idea
Exact p-values for a coin.
For n flips of a fair coin:
P(all n heads)=2n1;
P(at least n−1 heads)=2nn+1.
These are one-sided p-values for 'the coin favors heads'.
Worked example
10 flips, 9 or more heads.
1+10=11 out of 210=1024
Count the outcomes.
≈0.011: significant at the 5 % level
p-value.
Your turn, guided
Your turn: 5 flips, all heads: p-value as a fraction?
321
251.
Practice ★
A fair coin is flipped 11 times. Find the probability (as a fraction) that all 11 flips are heads.
A fair coin is flipped 7 times. Find the probability (as a fraction) that all 7 flips are heads.
A fair coin is flipped 10 times. Find the probability (as a fraction) that all 10 flips are heads.
Practice ★★
A fair coin is flipped 5 times. Find the p-value (as a fraction) for getting at least 4 heads.
A fair coin is flipped 7 times. Find the p-value (as a fraction) for getting at least 6 heads.
Exit ticket
Two questions to check.
A fair coin is flipped 11 times. Find the probability (as a fraction) that all 11 flips are heads.
A fair coin is flipped 7 times. Find the probability (as a fraction) that all 7 flips are heads.
Summary card
Coin p-values
All heads: 2n1.
At least n−1: 2nn+1.
Sign tests in medicine use exactly this idea: did more patients improve than chance allows?
8. History of significance tests: Arbuthnot, Pearson and Fisher
Discover
Records from 1629 to 1710 showed slightly more boys than girls born. In 1710 John Arbuthnot asked: could chance alone explain that? Suppose boys outnumbered girls in every one of those years.
Hardly: that would be like 82 heads in a row
Yes, easily
Key idea
Testing hypotheses, from baptisms to fields.
Step 1John Arbuthnot1667–1735Godfrey Kneller, 1723. Public domain1710: Arbuthnot's paper on male and female births, a first test of significance.
Step 2Karl Pearson1857–1936Unknown artist, 1910. Public domain1900: Pearson's chi-square test.
Step 3Ronald Fisher1890–1962Barry Eagel, 1952. Barry Eagel. CC BY-SA 4.01935: Fisher's The Design of Experiments.
In 1710 John Arbuthnot (1667–1735) studied the slight excess of male births over female births from 1629 to 1710, perhaps the first application of probability to social statistics and the first formal test of significance. In 1900 Karl Pearson published the chi-square test. At Rothamsted, Ronald Fisher developed the design and analysis of experiments.
Worked example
Arbuthnot's reasoning in numbers: how many years from 1629 to 1710, inclusive?
1710−1629+1=82
Suppose more boys every year: count the years.
(21)82≈2×10−25
Probability by chance alone.
Practice ★
What did Arbuthnot study in his 1710 paper?
The excess of male over female births
Fertilizers on randomized plots
The chi-square test
Like Arbuthnot: if each year were a fair 'coin flip' between more boys and more girls, what is the probability (as a fraction) of more boys 5 years in a row?
Like Arbuthnot: if each year were a fair 'coin flip' between more boys and more girls, what is the probability (as a fraction) of more boys 10 years in a row?
Pearson's chi-square: one category has expected count 10 and observed count 12. Compute its contribution E(O−E)2.
Summary card
Significance tests today
Medicine: does a new drug work better than a placebo?
Websites: does a new design get more clicks (A/B tests)?
Quality control: is a machine still making parts to specification?
Sources: MacTutor biographies of Arbuthnot, Pearson and Fisher (see "Sources and credits").
Answers: Significance tests
1. Stating hypotheses
A
a) 10021 b) 25004921 c) 103−25004921 d) 103 e) 2
The sample proportion p̂ (a statistic)
vary less around the population mean
B
H₀: p = 0.80
C
Hₐ: μ ≠ 81
D
Because we know p̂ exactly; the question is about the unknown population value
E
H₀: p = 0.80
H₀: p = 0.60
H₀: p = 0.62
F
100
2. The one-proportion z test and P-values
A
H₀: p = 0.80
Hₐ: μ ≠ 81
Because we know p̂ exactly; the question is about the unknown population value
B
0.4
1.6
C
4
D
If H₀ were true, a result at least as extreme as ours would happen with probability 0.05
E
0.4
1.6
1.2
F
10
3. Comparing two groups: two proportions and paired data
A
0.4
4
If H₀ were true, a result at least as extreme as ours would happen with probability 0.05
B
18443
9551
C
A paired t test on the 24 differences (after − before)
D
10042
E
18443
9551
20685
F
203
4. Conclusions, errors and power
A
0.4
4
If H₀ were true, a result at least as extreme as ours would happen with probability 0.05
B
Reject H₀: there is convincing evidence for Hₐ
C
Concluding the drug works when it really has no effect
D
Use a larger sample
E
Reject H₀: there is convincing evidence for Hₐ
Reject H₀: there is convincing evidence for Hₐ
Reject H₀: there is convincing evidence for Hₐ
F
1007
5. The one-sample t test for a mean
A
6.6048
9
Because σ is estimated by s, which adds variability
B
34
920
C
0.08
D
Fail to reject H₀: not enough evidence for Hₐ
E
34
920
1.875
F
1915
6. Chi-square tests
A
a) 201 b) 513 c) 53 d) 21 e) 59
Reject H₀: there is convincing evidence for Hₐ
Concluding the drug works when it really has no effect
B
10
C
7108
27121
D
1582255
671060
E
7108
27121
1
F
16
7. Exact p-values for a coin
A
20481
1281
10241
B
163
161
C
20481
1281
8. History of significance tests: Arbuthnot, Pearson and Fisher