1
Multiple comparisons – subsequent inferences for two-way ANOVA
• the kinds of inferences to be made after the F tests of a two-way ANOVA depend on the results
• if none of the F tests lead to rejection of the null hypothesis, then you have concluded that none of the means
are different and no further comparisons are required
Significant interactions
• if it has been determined that the interactions are significant, then the effect of each factor depends on the level
of the other factor
• another way of saying this is that each cell of the table has a possibly different mean µij
• there are IJ such means and r =
(
IJ
2
)
possible comparisons among them
• to control the overall type 1 error rate at α, the Bonferroni correction uses α∗ = α/r for each comparison
• confidence intervals have the form
ȳij. − ȳkl. ± tα∗/2,IJ(K−1)
√
MSE
√
2/K
• tests of H0 : µij = µkl versus Ha : µij 6= µkl are based on the statistic
tij,kl =
ȳij. − ȳkl.√
MSE
√
2/K
• the null hypothesis is rejected when P < α∗ or when |tij,kl| > tα∗/2,IJ(K−1)
• it is often easiest to rearrange this expression and reject H0 when
|ȳij. − ȳkl.| > tα∗/2,IJ(K−1)
√
MSE
√
2/K
• the right hand side remains constant so it is just a matter of looking at the differences between any two cell
means
Example: The following data are the lifetimes (in hours) of four different designs of an airplane wing subjected to
three different kinds of continuous vibrations.
Design
1 2 3 4
Vibration 1 876 1156 1234 825
913 1219 1181 797
Vibration 2 1413 1876 1591 1083
1290 1710 1649 1161
Vibration 3 1291 2115 1650 1148
1412 1963 1712 1262
2
• the ANOVA table is
Source SS DF MS F P
Vibration 1346145 2 673073 139.7
Design 1457096 3 485699 100.79
Interaction 138403 6 23067 4.79 .01
Error 57824 12 4819
Total 2999469 23
• the test of H0: no interactions versus Ha: there are interactions gives a p-value of .01, so we reject H0 testing
at the α = .05 level.
• to determine which combinations of vibration and design give significantly different results at we use the cell
means
Design
1 2 3 4
Vibration 1 894.5 1187.5 1207.5 811
Vibration 2 1351.5 1793 1620 1122
Vibration 3 1351.5 2039 1681 1205
• the cell means are significantly different if their absolute difference is greater than
tα∗/2,IJ(K−1)
√
MSE
√
2/K
• with I = 3, J = 4 and K = 2 there are r =
(
12
2
)
= 66 possible comparisons
• in this case MSE = 4819, α∗ = .05/66 = 0.000758 and tα∗/2,12 = 4.48, from minitab or R
• so the difference required for significance is
4.48 ∗
√
4819 ∗
√
2/2 = 310.73
3
• one way to do these comparisons efficiently is to rank them from smallest to largest,
using the notation (i, j) to represent the ith vibration and jth design
(i,j) mean mean + 310.73
(1,4) 811 1121.73
(1,1) 894.5 1205.23
(2,4) 1122 1432.73
(1,2) 1187.5 1498.23
(3,4) 1205 1515.73
(1,3) 1207.5 1518.23
(2,1) 1351.5 1662.23
(3,1) 1351.5 1662.23
(2,3) 1620 1930.73
(3,3) 1681 1991.73
(2,2) 1793 2103.73
(3,2) 2039
• the extra column shows the value required for a mean to be significantly different
• so (1,4) is not different from (1,1)
• (1,1) is not different from (2,4), (1,2) and (3,4)
• (2,4) is not different from (1,2), (3,4), (1,3), (2,1) and (3,1)
• (1,2) is not different from (3,4), (1,3), (2,1) and (3,1)
• (3,4) is not different from (1,3), (2,1) and (3,1)
• (1,3) is not different from (2,1) and (3,1)
• (2,1) is not different from (3,1) and (2,3)
• (3,1) is not different from (2,3)
• (2,3) is not different from (3,3) and (2,2)
• (3,3) is not different from (2,2)
• (2,2) is not different from (3,2)
• all other differences are significant
4
if the interactions were not significant
• if it is determined that the interactions are not significant then the main effects can be tested
• if both the row and column factors are significant then there are
r =
(
I
2
)
+
(
J
2
)
pairwise comparisons of interest
• if only the row factor or column factor is significant, r =
(
I
2
)
or r =
(
J
2
)
respectively
• in either case, the Bonferroni correction for multiple comparisons uses α∗ = α/r
• comparisons between the rows are made using row means, so that confidence intervals have the form
ȳi.. − ȳl.. ± tα∗/2,IJ(K−1)
√
MSE
√
2
JK
• (note that there are JK observations in each row)
• in comparing the rows we are making inferences about the difference in row effects αi − αl
• the test of H0 : αi − αl = 0 versus Ha : αi − αl 6= 0 uses
til =
ȳi.. − ȳl..
√
MSE
√
2
JK
• because the denominator is the same for all such statistics, one can simply compare the absolute difference
|ȳi..− ȳl..| to tα∗/2,IJ(K−1)
√
MSE
√
2
JK
and conclude the difference is significant if the former is larger than the
latter
• similarly, comparisons between the columns are made using column means, so that confidence intervals have the
form
ȳ.j. − ȳ.u. ± tα∗/2,IJ(K−1)
√
MSE
√
2
IK
• (note that there are IK observations in each column)
• in comparing the columns we are making inferences about the difference in column effects βj − βu
• the test of H0 : βj − βu = 0 versus Ha : βj − βu 6= 0 uses
tju =
ȳ.j. − ȳ.u.
√
MSE
√
2
IK
• because the denominator is the same for all such statistics, one can simply compare the absolute difference
|ȳ.j. − ȳ.u.| to tα∗/2,IJ(K−1)
√
MSE
√
2
IK
, and conclude the difference is significant if the former is larger than
the latter
5
Example: For the data on burn rates with 3 different engines and 4 different propellants, we determined that there
were no interactions but that both factors were significant, when testing at level α = .05.
• there are 3 comparisons to be made among the engines and 6 to be made among the propellants
• for an overall error rate of α = .05, the Bonferroni correction uses α∗ = .05/9 = 0.0056
• from the output from the model with interaction, MSE = 1.2425, with 12 degrees of freedom (Note that even
though the interaction was NOT significant, we use the MSE, and associated degrees of freedom, from the model
that included interaction.
• the critical t value is tα∗/2,12 = 3.37 using a computer program
• the engine means are ȳ1.. = 30.5, ȳ2.. = 29.675 and ȳ3.. = 28.60
• the required difference in engine means for significance is
tα∗/2,IJ(K−1)
√
MSE
√
2
JK
= 3.37
√
1.2425
√
2/8
= 3.37(1.1147)(.5)
= 1.8782
• using this approach we conclude that α1 is significantly different from α3, but that α1 is not different from α2
and α2 is not different from α3
• the propellant means are ȳ.1. = 31.6, ȳ.2. = 29.85, ȳ.3. = 28.38 and ȳ.4. = 28.53
• the required difference in propellant means for significance is
tα∗/2,IJ(K−1)
√
MSE
√
2
IK
= 3.37
√
1.2425
√
2/6
= 2.1689
• examining the propellant means shows that 3 and 4 are significantly different from 1, but that none of the other
comparisons are significant