The Holm-Šídák test in Prism
Prism can perform the Holm multiple comparisons test as part of several analyses:
•Following one-way ANOVA. This makes sense when you are comparing selected pairs of means, with the selection based on experimental design. Prism also lets you choose Bonferroni tests when comparing every mean with every other mean. We don't recommend this. Instead, choose the Tukey test if you want to compute confidence intervals for every comparison or the Holm-Šídák test if you don't.
•Following two-way ANOVA. If you have three or more columns, and wish to compare means within each row (or three or more rows, and wish to compare means within each column), the situation is much like one-way ANOVA. The Bonferroni test is offered because it is easy to understand, but we don't recommend it. If you enter data into two columns, and wish to compare the two values at each row, then we recommend the Bonferroni method, because it can compute confidence intervals for each comparison. The alternative is the Holm-Šídák method, which has more power, but doesn't compute confidence intervals.
•As part of the analysis that performs many t tests at once.
•To analyze a stack of P values.
•
Key facts about the Holm test
•The input to the Holm method is a list of P values, so it is not restricted to use as a followup test to ANOVA.
•The Holm multiple comparison test can calculate multiplicity adjusted P values, if you request them (2).
•The Holm multiple comparison test cannot compute confidence intervals for the difference between means.
•The method is also called the Holm step-down method.
• Although usually attributed to Holm, in fact this method was first described explicitly by Ryan (3) so is sometimes called the Ryan-Holm step down method.
•Holm's method has more power than the Bonferroni or Tukey methods (4). It has less power than the Newman-Keuls method, but that method is not recommended because it does not really control the familywise significance level as it should, except for the special case of exactly three groups (4).
•The Tukey and Dunnett multiple comparisons tests are used only as followup tests to ANOVA, and they take into account the fact that the comparisons are intertwined. In contrast, Holm's method can be used to analyze any set of P values, and is not restricted to use as a followup test after ANOVA.
•The Šídák modification of the Holm test makes it a bit more powerful, especially when there are many comparisons.
• Note that Šídák's name is used as part of two distinct multiple comparisons methods, the Holm-Šídák test and the Šídák test related to the Bonferroni test.
References:
1.Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 6 (2): 65–70.
2.Aickin, M. & Gensler, H. Adjusting for multiple testing when reporting research results: the Bonferroni vs Holm methods. American journal of public health 86, 726–728 (1996).
3.Ryan TA. Significance tests for proportions, variances, and other statistics. Psychol. Bull. 1960; 57: 318-28
4.MA Seaman, JR Levin and RC Serlin, New Developments in pairwise multiple comparisons: Some powerful and practicable procedures, Psychological Bulletin 110:577-586, 1991.
5.SA Glantz, Primer of Biostatistics, 2005, ISBN=978-0071435093.
