How to decide which P values are small enough to be deemed statistically significant

Here is a brief description of how the Holm multiple comparison test works:

1.If you are using the test as a followup to ANOVA, P values for each comparison are computed as they are for the Fisher's LSD test. These are not corrected for multiple comparisons.

2.The P values are ranked from smallest to largest.

3.Set a value for the significance level, alpha. This is often set to 5%.

4.Define K equal to the number of comparisons you are making.

5.Start with the smallest P value and set i=K. Ask: Is the smallest P value less than alpha/i?

If No: Conclude that none of the comparisons are statistically significant, and you are done.

If Yes: Conclude that this comparison is statistically significant, and continue.

6.The second to smallest P value is compared next. Set i=K-1. Is the P value less than alpha/i?

If No: Conclude that this  comparison (and all with larger P values) is not statistically significant. You  are done.

If Yes: Conclude that this comparison is statistically significant, and continue.

7.The third to smallest P value is compared next. Set i=K-2. Compare the P value to alpha/i...

8.Continue until you find a comparison that is not statistically significant.

Prism actually uses the Šídák modification, so computes the Holm-Šídák test. At steps 5-7 above, the P value is not compared to to alpha/i but rather to 1-(1-alpha)(1/i)

Multiplicity adjusted P values from the Holm-Sidak method

To compute the adjusted P value, called Padj(i), from the P value, called P(i), use the equations below where, the P values are sorted so P(1) is the smallest,  k is the number of comparisons (number of P values) and max is a function that returns the larger of two values.

PAdj(1) = 1 - (1 - P(1))^k

PAdj(2) = max(PAdj(1), 1 - (1 - P(2))^(k-1))

..........

PAdj(j) = max(PAdj(j-1), 1 - (1 - P(j))^(k-j+1))

..........

PAdj(k) = max(PAdj(k-1), 1 - (1 - P(k))^(k-k+1)) = max(PAdj(k-1), P(k))

 

Note that in some cases successive adjusted P values will be identical, even when the original P values are not.