The normality test will give an indication of whether the populations from which the samples were drawn appear to be normally distributed, but will not indicate the cause(s) of the nonnormality. The smaller the sample size, the less likely the normality test will be able to detect nonnormality.
Although the goal of each multiple comparison test is to keep the overall significance level at the desired value, some multiple comparison tests are more conservative (less powerful) than others. In general, for comparing all pairs of means, Scheffé's test will be more conservative than the Tukey test, which in turn will be more conservative than the Newman-Keuls test. This means, for example, that a pair of means might be flagged as significantly different by the Newman-Keuls test, but not by Scheffé's test when performed on the same data. The Bonferroni method may be the most powerful when only a few specific mean differences are to be tested, but when used as a test of all possible mean differences, it quickly loses power relative to the other all-pairwise tests as the number of groups grows.
Because the one-way ANOVA F test is often a more powerful test than a multiple comparisons test, it is possible for the F test to reject the null hypothesis that the means are equal, while the multiple comparison test does not show any significantly different pairs of means. This is more likely to happen when the sample sizes are small.
If we use a multiple comparisons test to divide the means into subgroupings, the test may produce ambiguous results. For example, a test involving three samples, ordered from lowest mean to highest mean, may conclude that mean 1 is different from mean 3, but that mean 2 is not different from either mean 1 or mean 3. This suggests that there are two groups of means, but we can not decide from the test to which group mean 2 belongs. This problem is usually due to lack of power (often from small sample sizes).
By keeping the overall significance level at the desired value, multiple comparisons tests limit the probability of incorrectly flagging one or more pairs of means as being significantly different. However, if a multiple comparisons test incorrectly flags a pair of means as significantly different, the probability of then making a second such mistake is much more than the desired significance level. If a number of mean pairs are unexpectedly flagged as significantly different, this may be the reason.
The normality test will give an indication of whether the populations from which the samples were drawn appear to be normally distributed, but will not indicate the cause(s) of the nonnormality. The smaller the sample size, the less likely the normality test will be able to detect nonnormality.
A fan pattern like the profile of a megaphone, with a noticeable flare either to the right or to the left as shown in the picture (one or more of the "stacks" of residuals is much longer than the others), often suggests that the variance in the values increases in the direction the fan pattern widens (often to the right), and this in turn suggests that a transformation may be needed. Other systematic pattern in the residuals (like a linear trend) suggest either that there is another factor that should be considered in analyzing the data, or that a transformation is needed.
Back
Back