The calculations use a pooled SD

The first step for the Bonferroni and Sidak tests used as a followup to ANOVA is to compute the Fisher LSD test. Note two important points:

•The P values from this test are not corrected for multiple comparisons, so the correction for multiple comparisons is done as a second step.

•The P values are computed from difference between the two means being compared and the overall pooled SD. When you compare columns A and B, the values in columns C, D, E, etc. affect the calculation of the pooled SD so affect the P value for the comparison of A and B. Using a pooled SD makes sense if all the values are sampled from populations with the same SD, as use of the pooled SD gives the Bonferroni or Sidak test more degrees of freedom, and therefore more power.

How the Šídák multiple comparison test works

The logic is simple(1). If you perform three independent comparisons (with the null hypothesis actually true for each one), and use the conventional significance threshold of 5% for each comparison without correcting for multiple comparisons, what is the chance that one or more of those tests will be declared to be statistically significant? The best way to approach that question, is to ask the opposite question -- what is the chance that all three comparisons will reach a conclusion that the differences are not statistically significant? The chance that each test will be not significant is 0.95, so the chance that all three independent comparisons will be not statistically significant is 0.95*0.95*0.95, which equals 0.8574.  Now switch back to the original question. The chance that one or more of the comparisons will be statistically significant is 1.0000 - 0.8574, which is 0.1426.

You can also start with the significance threshold that you want to apply to the entire family of comparisons, and use the Šídák-Bonferroni method to compute the significance threshold that you must use for each individual comparison.

Call the significance threshold for the family of comparisons, the familywise alpha, alphaFW, and the number of comparisons K. The significance threshold to use for each individual comparisons, the per comparison alpha (alphaPC), is defined to be:

alphaPC = 1.0  -  (1.0 - alphaFW)1/K

If you are making three comparisons, and wish the significance threshold for the entire family to be 0.05, then the threshold for each comparison is:

alphaPC = 1.0  -  (1.0 - alphaFW)1/K = 1.0 - (1.0 - 0.05)1/3= 0.0170

If you are making ten comparisons, and wish the significance threshold for the entire family of comparisons to be 0.05, then the threshold for each comparison is:

alphaPC = 1.0  -  (1.0 - alphaFW)1/K = 1.0 - (1.0 - 0.05)0.10=  0.0051

How the Bonferroni multiple comparison test works

The Bonferroni method uses a simpler equation to answer the same questions as the Šídák method. If you perform three independent comparisons (with the null hypothesis actually true for each one), and use the conventional significance threshold of 5% for each one without correcting for multiple comparisons, what is the chance that one or more of those tests will be declared to be statistically significant?

The Bonferroni method simply multiplies the individual significance threshold (0.05) by the number of comparisons (3), so the answer is 0.15. This is close, but not the same as the more accurate calculations above, which computed the answer to be 0.1426. (With many comparisons, the product of the significance threshold times the number of comparisons can exceed 1.0; in this case, the result is reported as 1.0.)

To use the Bonferroni method to compute the significance threshold to use for each comparison (alphaPC) from the number of comparisons and the significance threshold you wish to apply to the entire family of comparisons (alphaFW), use this simple equation:

alphaPC = alphaFW/K

Let's say you set the significance threshold for the entire family of comparisons to 0.05 and that you are making three comparisons. The threshold for determining significance for any particular comparison is reduced to 0.05/3, or 0.0167. Note that this is a bit more strict than the result computed above for the Šídák method, 0.0170.

If you are making ten comparisons, the Bonferroni threshold for each comparisons is 0.05/10 = 0.0050. Again this is a bit more strict (smaller) than the value computed by the Šídák method above, which is 0.0051.