Do multiple-comparison tests following one-way ANOVA always have less power than a t test?
No.
Post tests control for multiple comparisons. The significance level doesn't apply to each comparison, but rather to the entire family of comparisons. In general, this makes it is 'harder' to conclude that a difference is statistically significant. This is really the main point of multiple comparisons, as it reduces the chance of being fooled by a differences that are due entirely to random sampling.
But multiple comparisons tests do more than set a stricter threshold of significance. They also use the information from all of the groups, even when comparing just two. It uses the information in the other groups to quantify variation more precisely. Since the scatter is determined from more data, there are more degrees of freedom in the calculations, and this usually offsets some of the increased strictness mentioned above.
In some cases, the effect of increasing the df overcomes the effect of controlling for multiple comparisons. In these cases, you may find a multiple comparisons test might lead to a conclusion that a difference is statistically significant even though a simple t test concludes that the difference is not statistically significant.
See an example below. We'll defined the significance level to be 5%, so only P values less than 0.05 are considered statistically significant. If you compare groups A and B by unpaired t test, the two-tailed P value equals 0.0557, so the results are not 'statistically significant' by the threshold we established. But if you compare all four groups with one-way ANOVA, and follow with Tukey multiple comparison tests of every pair, the difference between groups A and B is statistically significant at the 0.05 significance level.
Why the discrepancy? Because ANOVA is based on pooling the scatter from all our groups. In this example, groups C and D are more consistent than are groups A and B. The extra scatter in A and B push up the P value in the t test, compared to the Tukey multiple comparison test.
A | B | C | D |
34. | 37. | 48. | 52. |
32. | 43. | 49. | 51. |
29. | 35. | 48. | 52. |
37. | 40. | 48. | 53. |
Download the Prism file from this example.
(This example was completely revised in Sept. 09. The previous example was incorrect, and didn't actually demonstrate the point.)