Viewing By Entry / Main
August 17, 2009
Adjusted P values as part of multiple comparisons.

 Many people ask why multiple comparisons tests following one-way (or two-way) ANOVA can't report individual P values for each comparison. When you correct for multiple comparisons, it really doesn't make much sense to talk about individual P values. All you can do is divide the comparisons into two groups -- statistically significant and not -- at some defined significance level (usually 5%) that applies to the entire family of comparisons. 

However SAS reports adjusted P values, and these are explained in the book by Westfall  (citation below).

The idea is pretty simple. There is nothing special about 0.05 or 0.01... You can set the significance level to any probability you want. The adjusted P value is the smallest probability at which a particular comparison will be declared statistically significant (as part of the multiple comparison testing). Each comparison will have a unique adjusted P value. But these P values are computed from all the comparisons, and really can't be interpreted for just one comparison. 

Computing the adjusted P value is trivial for Bonferroni multiple comparison tests. It is harder for Tukey and Dunnett tests, as it requires computing critical values beyond those that have been tabulated. 

We are considering including adjusted P values as part of the reporting of one-way ANOVA with Tukey, Bonferroni or Dunnett multiple comparison tests. I hesitate to do this (the programming is far from trivial) because I don't really know how to explain how to interpret the results, and fear that they will be misinterpreted. 

I don't know of any program except SAS that computes adjusted P values, and don't know of any book except Westfall that explains them. They do not seem to be mainstream. 

Please let us know if you would like to see adjusted P values in future versions of Prism, and explain why.

Multiple Comparisons and Multiple Tests (Text and Workbook Set) Multiple Comparisons and Multiple Tests (Text and Workbook Set)
by Peter H. Westfall, Randall D. Tobias, Dror Rom
IBSN:1580258336. List price:$62.32
Buy from amazon.com for $50.18

Comments

There are no comments for this entry.