Lessons and exercises follow the school system and stay in its language.Les leçons et les exercices suivent le système scolaire et restent dans sa langue.Lektionen und Übungen folgen dem Schulsystem und bleiben in seiner Sprache.

Inference for regression

Print the chapter

Every lesson of the chapter, ready for paper. Exercise sets are lettered A, B, C… and answered at the end.

To save a PDF, choose “Save as PDF” in the print dialog.

1. The regression model and its conditions

Warm-up

Three quick questions from earlier lessons.

  1. A company claims that 80% of customers are satisfied. A researcher suspects fewer are. Which is the null hypothesis?
    1. H₀: p = 0.80
    2. H₀: p < 0.80
    3. H₀: p̂ = 0.80
  2. The mean wait at a clinic used to be 81 minutes. A manager wants to know whether it has changed, in either direction. Which alternative?
    1. Hₐ: μ ≠ 81
    2. Hₐ: μ > 81
    3. Hₐ: x̄ ≠ 81
  3. Why are hypotheses written about parameters such as p, not statistics such as p̂ (sample of 445)?
    1. Because we know p̂ exactly; the question is about the unknown population value
    2. Because statistics are harder to compute
    3. There is no difference between p and p̂

Discover

Two random samples of 30 students give regression slopes 2.1 and 2.6 for hours studied vs score. Why do they differ?

  1. Sampling variability: each sample estimates the same true slope
  2. One of the samples must be wrong

Key idea

A model for the population.

Population regression line: μy=β0+β1x\mu_y = \beta_0 + \beta_1 x, estimated by y^=a+bx\hat{y} = a + bx. Conditions for inference (LINER): Linear relationship, Independent observations, Normal residuals, Equal spread of residuals, Random sample or assignment.

Worked example

Checking the conditions with a residual plot.

Linear: residuals scattered around 0
No curve.
Equal spread across xx
No fan.
Roughly normal: no strong skew or outliers
Histogram of residuals.

Your turn, guided

Your turn: how many conditions are in LINER?

55
L, I, N, E, R.

Practice ★

  1. A regression on a random sample of 23 houses gives slope b. What does b estimate?
    1. The slope β₁ of the population regression line
    2. Nothing: b is the true slope
    3. The correlation r
  1. Which conditions are needed for inference about the slope from 75 pairs? Tick all that are.
    1. The true relationship is linear
    2. Observations are independent (random sample, 10% condition)
    3. Residuals are roughly normal
    4. Residuals have about equal spread for all x
    5. The correlation is above 0.5

Watch out!

Practice ★★

  1. The residual plot of a regression on 85 points fans out: residuals spread more as x grows. Which condition fails?
    1. Equal variance of the residuals
    2. Linearity
    3. Normality of the population

Exit ticket

Three questions to check you have got it.

  1. A regression on a random sample of 23 houses gives slope b. What does b estimate?
    1. The slope β₁ of the population regression line
    2. Nothing: b is the true slope
    3. The correlation r
  2. A regression on a random sample of 59 houses gives slope b. What does b estimate?
    1. The slope β₁ of the population regression line
    2. Nothing: b is the true slope
    3. The correlation r
  3. A regression on a random sample of 46 houses gives slope b. What does b estimate?
    1. The slope β₁ of the population regression line
    2. Nothing: b is the true slope
    3. The correlation r

Summary card

The regression model

y^=a+bx\hat{y} = a + bx estimates μy=β0+β1x\mu_y = \beta_0 + \beta_1 x. Conditions: LINER.

Watch out: no condition on the size of rr.

One more step is optional: a challenge. Try it, or finish the lesson now.

Challenge ★★★

  1. The regression model gives ŷ = 17 + 0.4x. By how much does the predicted y change when x increases by 10?

2. The t statistic for the slope

Warm-up

Three quick questions from earlier lessons.

  1. A regression on a random sample of 23 houses gives slope b. What does b estimate?
    1. The slope β₁ of the population regression line
    2. Nothing: b is the true slope
    3. The correlation r
  2. Which conditions are needed for inference about the slope from 75 pairs? Tick all that are.
    1. The true relationship is linear
    2. Observations are independent (random sample, 10% condition)
    3. Residuals are roughly normal
    4. Residuals have about equal spread for all x
    5. The correlation is above 0.5
  3. The residual plot of a regression on 85 points fans out: residuals spread more as x grows. Which condition fails?
    1. Equal variance of the residuals
    2. Linearity
    3. Normality of the population

Discover

A sample slope is b = 0.6 with standard error 0.2. How many standard errors is it from 0?

  1. 3
  2. 0.12

Key idea

Standardize the slope.

t=b−β1(0)SEbt = \dfrac{b - \beta_1^{(0)}}{SE_b}, usually with β1(0)=0\beta_1^{(0)} = 0, and df = n−2n - 2. Computer output lists bb (Coef), SEbSE_b (SE Coef), tt and the P-value in the row of xx.

Worked example

Output for xx: Coef 1.25, SE Coef 0.50, n=22n = 22.

t=2.5t = 2.5
1.25/0.50.
df =20= 20
22 − 2.
P-value ≈0.021\approx 0.021
Two-sided, from the t distribution.

Your turn, guided

Your turn: b = 0.9, SE_b = 0.3. Find t.

t=3t = 3
0.9/0.3.

Practice ★

  1. Computer output gives slope b = 8/10 with standard error SE_b = 3/100. Find the t statistic for H₀: β₁ = 0.
  2. Computer output gives slope b = 9/10 with standard error SE_b = 8/100. Find the t statistic for H₀: β₁ = 0.
  1. A regression uses 20 data pairs. How many degrees of freedom does the t test for the slope have?

Watch out!

Practice ★★

  1. In regression output, the row for x reads: Coef 0.97, SE Coef 0.08, and a t value. Which number is the slope's standard error?
    1. The SE Coef in the x row
    2. The Coef in the x row
    3. The value s (the SD of the residuals)

Exit ticket

Three questions to check you have got it.

  1. Computer output gives slope b = 8/10 with standard error SE_b = 3/100. Find the t statistic for H₀: β₁ = 0.
  2. Computer output gives slope b = 9/10 with standard error SE_b = 8/100. Find the t statistic for H₀: β₁ = 0.
  3. Computer output gives slope b = 7/10 with standard error SE_b = 3/100. Find the t statistic for H₀: β₁ = 0.

Summary card

t for the slope

t=b/SEbt = b/SE_b, df =n−2= n - 2.

Watch out: divide by SEbSE_b; two parameters, so n−2n - 2.

One more step is optional: a challenge. Try it, or finish the lesson now.

Challenge ★★★

  1. Output gives b = 3/10 with SE_b = 4/100. With a larger sample, SE_b is halved and b stays the same. Find the new t statistic for H₀: β₁ = 0.

3. Confidence interval for the slope

Warm-up

Three quick questions from earlier lessons.

  1. A random sample of 25 has standard deviation s = 16. With t* = 2.064 (24 df, 95%), find the margin of error for the mean.
  2. A one-sample t interval uses a sample of 10. How many degrees of freedom?
  3. Why is a t critical value used for a confidence interval for a mean from a sample of 40?
    1. Because σ is estimated by s, which adds variability
    2. Because n is less than 1,000
    3. Because the data are skewed

Discover

b = 0.5, SE_b = 0.1, t* = 2. Which interval estimates β₁?

  1. (0.3, 0.7)
  2. (0.4, 0.6)

Key idea

An interval for the population slope.

b±t∗⋅SEbb \pm t^* \cdot SE_b, with t∗t^* from the t distribution with n−2n - 2 df. If the interval does not contain 0, there is convincing evidence of a linear relationship; interpret as the change in mean yy per unit increase in xx.

Worked example

n=30n = 30, b=0.42b = 0.42, SEb=0.10SE_b = 0.10, t∗=2.048t^* = 2.048.

ME=0.2048ME = 0.2048
2.048 × 0.10.
(0.215,0.625)(0.215, 0.625)
0.42 ± 0.2048.
Evidence of a positive slope
0 excluded.

Your turn, guided

Your turn: ME = 0.15. The width of the interval.

0.30.3
2 × 0.15.

Practice ★

  1. With 30 pairs, b = 2/10 and SE_b = 7/100. Using t* = 2.048 (28 df, 95%), find the upper end of the interval for β₁.
  2. With 30 pairs, b = 4/10 and SE_b = 1/100. Using t* = 2.048 (28 df, 95%), find the upper end of the interval for β₁.
  1. SE_b = 1/100 and t* = 2.048. Find the width of the 95% confidence interval for β₁.

Watch out!

Practice ★★

  1. A 95% interval for the slope β₁ is (0.4, 0.8). What does it suggest?
    1. Convincing evidence of a positive linear relationship: 0 is not in the interval
    2. No evidence of a relationship
    3. x causes y to increase

Exit ticket

Three questions to check you have got it.

  1. With 30 pairs, b = 2/10 and SE_b = 7/100. Using t* = 2.048 (28 df, 95%), find the upper end of the interval for β₁.
  2. With 30 pairs, b = 4/10 and SE_b = 1/100. Using t* = 2.048 (28 df, 95%), find the upper end of the interval for β₁.
  3. With 30 pairs, b = 3/10 and SE_b = 6/100. Using t* = 2.048 (28 df, 95%), find the upper end of the interval for β₁.

Summary card

Interval for β₁

b±t∗SEbb \pm t^* SE_b (df n−2n - 2); 0 outside → evidence of a relationship.

Watch out: association, not causation.

One more step is optional: a challenge. Try it, or finish the lesson now.

Challenge ★★★

  1. A 95% confidence interval for the slope is (0.2, 0.5). What is the point estimate b?

4. Testing the slope and reading output

Warm-up

Three quick questions from earlier lessons.

  1. Computer output gives slope b = 8/10 with standard error SE_b = 3/100. Find the t statistic for H₀: β₁ = 0.
  2. A regression uses 20 data pairs. How many degrees of freedom does the t test for the slope have?
  3. In regression output, the row for x reads: Coef 0.97, SE Coef 0.08, and a t value. Which number is the slope's standard error?
    1. The SE Coef in the x row
    2. The Coef in the x row
    3. The value s (the SD of the residuals)

Discover

If x is no help in predicting y, what is the population slope?

  1. 0
  2. 1

Key idea

Test whether the slope is zero.

H0:β1=0H_0: \beta_1 = 0 (no linear relationship), Ha:β1≠0H_a: \beta_1 \ne 0 (or one-sided). Use t=b/SEbt = b/SE_b, df =n−2= n - 2. A small P-value is convincing evidence of a linear relationship in the population. Read bb, SEbSE_b, tt and P from the output.

Worked example

Output: b = 0.30, t = 6.0, P < 0.001.

SEb=0.30/6=0.05SE_b = 0.30/6 = 0.05
t = b/SE_b.
Reject H₀
Very small P.
Convincing evidence of a linear relationship
In context.

Your turn, guided

Your turn: b = 0.8 and t = 4. Find SEbSE_b.

0.20.2
0.8/4.

Practice ★

  1. To test whether x is useful for predicting y (from 51 pairs), what is the null hypothesis?
    1. H₀: β₁ = 0, no linear relationship in the population
    2. H₀: b = 0
    3. H₀: r = 1
  1. Output shows slope b = 6/10 and t = 9 for H₀: β₁ = 0. Find SE_b.
  2. Output shows slope b = 3/10 and t = 9 for H₀: β₁ = 0. Find SE_b.

Watch out!

Practice ★★

  1. An observational study tests H₀: β₁ = 0 for hours of sleep (x) and test score (y). The P-value is 0.006. What can you conclude?
    1. Convincing evidence of a linear association between sleep and scores in the population sampled
    2. More sleep causes higher scores
    3. The correlation must be close to 1

Exit ticket

Three questions to check you have got it.

  1. Output shows slope b = 6/10 and t = 9 for H₀: β₁ = 0. Find SE_b.
  2. Output shows slope b = 3/10 and t = 9 for H₀: β₁ = 0. Find SE_b.
  3. Output shows slope b = 3/10 and t = 6 for H₀: β₁ = 0. Find SE_b.

Summary card

Testing the slope

H₀: β₁ = 0; t=b/SEbt = b/SE_b, df =n−2= n - 2.

Watch out: a significant slope is not proof of causation.

One more step is optional: a challenge. Try it, or finish the lesson now.

Challenge ★★★

  1. The t statistic for H₀: β₁ = 0 leaves an area of 0.008 in one tail. Find the two-sided P-value.

5. Slope and r² from r

Discover

The regression line of height on arm span has r=0.8r = 0.8. What part of the variation in height does the line explain?

  1. r2=0.64r^2 = 0.64, about 64 %
  2. 0.80.8, 80 %

Key idea

Slope and r2r^2 from rr.

For the least-squares line of yy on xx:

b=r sysxb = r\,\dfrac{s_y}{s_x} \quad and \quad r2r^2 = fraction of the variation in yy explained by xx.

Worked example

r=0.6r = 0.6, sx=5s_x = 5, sy=10s_y = 10.

b=0.6×105=1.2b = 0.6 \times \dfrac{10}{5} = 1.2
Slope.
r2=0.36r^2 = 0.36: 36 %
Explained.

Your turn, guided

Your turn: r=0.5r = 0.5, sx=2s_x = 2, sy=8s_y = 8. Find bb.

b=2b = 2
0.5×820.5 \times \dfrac{8}{2}.

Practice ★

  1. r=15r = \frac{1}{5}, sx=9s_x = 9, sy=27s_y = 27. Find the slope of the regression line of yy on xx.
  2. r=15r = \frac{1}{5}, sx=1s_x = 1, sy=9s_y = 9. Find the slope of the regression line of yy on xx.
  3. r=45r = \frac{4}{5}, sx=6s_x = 6, sy=48s_y = 48. Find the slope of the regression line of yy on xx.

Practice ★★

  1. A regression has r=−35r = -\frac{3}{5}. What percent of the variation in yy is explained by xx?
  2. A regression has r=−710r = -\frac{7}{10}. What percent of the variation in yy is explained by xx?

Exit ticket

Two questions to check.

  1. r=15r = \frac{1}{5}, sx=9s_x = 9, sy=27s_y = 27. Find the slope of the regression line of yy on xx.
  2. r=15r = \frac{1}{5}, sx=1s_x = 1, sy=9s_y = 9. Find the slope of the regression line of yy on xx.

Summary card

r, slope and r²

  • b=r sysxb = r\,\dfrac{s_y}{s_x}.
  • r2r^2 is the share explained.

Computer output shows 'R-sq': it is this r2r^2, as a percent.

6. History of regression: Galton and Pearson

Discover

Very tall parents tend to have tall children, but usually a bit less tall. In the 1880s, Francis Galton gave this a name. Which?

  1. Regression toward the mean
  2. Correlation

Key idea

From heredity to inference.

182018401860188019001822Galton1822Galton
  1. Step 1182018401860188019001822Galton1822GaltonFrancis Galton 1822–1911Eveleen Myers (née Tennant), 1890s. Public domainGalton (born 1822) names regression to the mean.
  2. Step 2182018401860188019001822Galton1857Pearson1822Galton1857PearsonKarl Pearson 1857–1936Unknown artist, 1910. Public domainKarl Pearson (1857–1936): regression analysis and the correlation coefficient.
  3. Step 3182018401860188019001822Galton1857Pearson1901the journal Biometrika1822Galton1857Pearson1901the journal Biometrika1900: the chi-square test; October 1901: first volume of Biometrika.

Francis Galton (born 1822) invented the idea of 'regression to the mean', first calling it 'reversion'. Karl Pearson (1857–1936) contributed to regression analysis and the correlation coefficient, and in 1900 the chi-square test. With Weldon and Galton he founded the journal Biometrika, first published in October 1901.

Worked example

Galton's idea: slope 0.6, parents 10 cm above the mean height.

0.6×10=60.6 \times 10 = 6 cm above the mean
Predicted child.
regression
Closer to the mean.

Practice ★

  1. What did Galton first call regression to the mean?
    1. Reversion
    2. Correlation
    3. Least squares
  1. Galton's line has slope 0.4. A father is 11 cm above the mean. How many cm above the mean is his son predicted to be?
  2. Galton's line has slope 0.7. A father is 18 cm above the mean. How many cm above the mean is his son predicted to be?
  1. A regression line is y^=12+0.6x\hat{y} = 12 + 0.6x. Predict yy for x=8x = 8.

Summary card

Regression today

  • Economists test whether a slope is really different from 0.
  • Medicine: does a dose change blood pressure?
  • Sports analytics: performance regresses toward the mean.

Sources: MacTutor biographies of Galton and Pearson (see "Sources and credits").