1
Multiple comparisons – subsequent inferences for one-way ANOVA
• if the overall F test does not show significant differences among the groups, then
no further inferences are required
• if the overall test of equality of means concludes in favour of the alternative HA :
not all means are the same, then the natural question is “which of the means
are difference”
The differences between a particular means, say of the i’th and k’th populations
can be tested with a t-test, using the test statistic
T =
x̄i. − x̄k.
√
MSE
√
1
ni
+ 1
nk
• The null hypothesis here is H0,ik : µi = µk and the alternative is HA,ik : µi 6= µk.
• in this expressions, MSE is the estimate of σ2 from the analysis of variance
• the degrees of freedom for the t-test is N − a, which is the degrees of freedom
associated with MSE in the ANOVA
• adjustment must be made for the fact that we are doing multiple
comparisons, that is, for the fact that several tests are being done, sometimes
known as simultaneous inference
• the simplest adjustment is the Bonferroni correction, which reduces the signif-
icance level for each test so that the overall probability of making at least one
type I error is no larger than the level α associated with the ANOVA
• in a one-way ANOVA with a groups, there are r =
a
2
natural comparisons
between pairs of groups
• if you do r tests each at level α, then the probability of incorrectly rejecting at
least one null hypothesis could be as large as rα
– for example for r = 2
P (reject at least one H0) =
P (reject 1st) + P (reject 2nd)
−P (reject both) ≤ 2α
• to control the overall level, or experimentwise error rate, at α, each test should
be done using α∗ = α/r
2
• alternatively the P value should be multiplied by r
• similarly for r confidence intervals, use of α∗ will give simultaneous confidence
level 1− α
• These simultaneous confidence intervals for the differences of two means are of
the form
x̄i. − x̄k. − tα∗/2,N−a√MSE
√√√√ 1
ni
+
1
nk
, x̄i. − x̄k. + tα∗/2,N−a
√
MSE
√√√√ 1
ni
+
1
nk
3
Example: for the golf balls, the summary statistics are
i x̄i s
2
i ni
1 251.28 33.487 5
2 261.98 18.197 5
3 269.66 27.253 5
• the value for MSE is 26.312
• there are 3 possible pairwise comparisons between the groups
• the denominator of the test statistics is
√
MSE
√√√√ 1
ni
+
1
nk
= 5.1295(.6325) = 3.24
• the degrees of freedom are 12
• with α = .05, α∗ = .05/3 = .0167, α∗/2 = .00833, and t.00833,12 = 2.7794,
• found, for example, as
MTB > invcdf .00833;
SUBC> t 12.
Inverse Cumulative Distribution Function
Student’s t distribution with 12 DF
P(X<=x) x 0.00833 -2.77969 • the test statistics are t12 = 251.18− 261.98 3.24 = −3.329 t13 = 251.18− 269.66 3.24 = −5.697 and t23 = 261.18− 269.66 3.24 = −2.367 • the first two comparisons are significant at the .05 level but the third one is not • confidence intervals for the differences in means are −10.8± 2.78(3.24) or (−19.81,−1.79) −18.48± 9.01 or (−27.49,−9.47) and −7.68± 9.01 or (−16.69, 1.33) 4 Example: for the liver weights, the means in ascending order are diet B C A D ni 8 6 7 8 mean 3.43 3.598 3.803 3.935 • the estimated standard deviation is√ MSE = .1899 • there are 6 comparisons, so the appropriate table value for α = .05 is t .025/6 25 = 2.8649, from MINITAB • the pairwise differences in the means are i/k B C D A .373 .205 -.132 B -.168 -.505 C -.337 • the absolute difference in means must exceed t.025/625 √ MSE √ 1 ni + 1 nk , which depends on the two sample sizes. ni/nk 7 8 6 .3025 .2938 7 .2818 8 .2720 • using this table, we find that B and C, C and A and A and D are not statistically significant, the other 3 comparisons are significant