The Holm-Šídák test in Prism
Prism can perform the Holm multiple comparisons test as part of several analyses:
•As a multiple corrections test following ANOVA
oFollowing one-way ANOVA
This makes sense when you are comparing selected pairs of means, with the selection based on experimental design. Prism also lets you choose Bonferroni tests when comparing every mean with every other mean. We don't recommend this. Instead, choose the Tukey test if you want to compute confidence intervals for every comparison or the Holm-Šídák test if you don't.
oFollowing two-way ANOVA
If you have three or more columns of data, you may choose to to compare means within each row (or with three or more rows to compare means within each column). In this case, the situation is much like one-way ANOVA: the Bonferroni test is offered because it is easy to understand, but we don't recommend it. Instead, we suggest choosing the Tukey test if you want to compute confidence intervals for every comparison or the Holm-Šídák test if you don't.
If your data are only entered into two columns, you may choose to compare the two values at each row (or with two rows to compare the two values within each column). In this case, we recommend the Bonferroni method as it can compute confidence intervals for each comparison. The alternative is to use the Holm-Šídák method, which has more power but which doesn't compute confidence intervals.
•As part of the analysis that performs many t tests at once.
•To analyze a stack of P values.
Key facts about the Holm test
•The input to the Holm method is a list of P values, so it is not restricted to use as a followup test to ANOVA.
•The Holm multiple comparison test can calculate multiplicity adjusted P values, if you request them (2).
•The Holm multiple comparison test cannot compute confidence intervals for the difference between means.
•The method is also called the Holm step-down method.
• Although usually attributed to Holm, in fact this method was first described explicitly by Ryan (3) so is sometimes called the Ryan-Holm step down method.
•Holm's method has more power than the Bonferroni or Tukey methods (4). It has less power than the Newman-Keuls method, but that method is not recommended because it does not really control the familywise significance level as it should, except for the special case of exactly three groups (4).
•The Tukey and Dunnett multiple comparisons tests are used only as followup tests to ANOVA, and they take into account the fact that the comparisons are intertwined. In contrast, the Holm method can be used to analyze any set of P values, and is not restricted to use as a followup test after ANOVA. In Prism, analyzing a stack of P values can be performed using the Holm-Šídák, Bonferroni-Dunn (often, just "Dunn"), or Bonferroni-Šídák (often, just "Šídák") methods.
•The Šídák modification of the Holm test mentioned above makes it a bit more powerful, especially when there are many comparisons.
• Note that Šídák's name is used as part of two distinct multiple comparisons methods, the Holm-Šídák test and the Šídák test related to the Bonferroni test.
References:
1.Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 6 (2): 65–70.
2.Aickin, M. & Gensler, H. Adjusting for multiple testing when reporting research results: the Bonferroni vs Holm methods. American journal of public health 86, 726–728 (1996).
3.Ryan TA. Significance tests for proportions, variances, and other statistics. Psychol. Bull. 1960; 57: 318-28
4.MA Seaman, JR Levin and RC Serlin, New Developments in pairwise multiple comparisons: Some powerful and practicable procedures, Psychological Bulletin 110:577-586, 1991.
5.SA Glantz, Primer of Biostatistics, 2005, ISBN=978-0071435093.
