Bug Fixed: Confidence Intervals for the Dunnett T3 multiple comparisons test reported incorrectly prior to Prism 9.3.1

Note: Prism 9.3.1 was released on December 16, 2021 and includes the fix for this bug.

The issue

In versions of Prism prior to 9.3.1, there was a problem with the confidence intervals reported along with the Dunnett T3 test. When correcting for multiple comparisons using the Dunnett T3 test, the confidence intervals that Prism reported were too wide (too conservative) given the data. Note, however, that the adjusted P value that Prism reported along with these confidence intervals was correct. This means that in a small number of cases, a P value of less than 0.05 may have been reported along with a 95% confidence interval that incorrectly included zero in its reported range.

Note that this issue only applied to the Dunnett T3 test, and this multiple comparisons test is only available when performing a Brown-Forsythe and Welch ANOVA (part of Prism’s One-Way ANOVA options when the standard deviations or variances of groups are not assumed to be equal). This issue did not affect the standard Dunnett’s test, nor did it affect multiple comparisons tests for two-way or three-way ANOVA.
 

Results in versions prior to Prism 9.3.1

Corrected results beginning in version 9.3.1

A bit more detail

This issue was present in all versions of Prism that provided the Dunnett T3 multiple comparisons test prior to 9.3.1. This test is provided as the default option for multiple comparisons when performing a Brown-Forsythe and Welch ANOVA (assuming unequal SD for each group), and multiple comparisons of every group compared to a single control group.

Without getting into the mathematical details of the test itself, the easiest explanation as to the source of this issue was the way that Prism was using the alpha (α) threshold level. This value can be specified on the Options tab of the “Parameters: One-Way ANOVA (and Nonparametric or Mixed)” analysis dialog, and - by default - is set to a value of 0.05 (corresponding to a 95% confidence level). However, prior to version 9.3.1, Prism incorrectly reported confidence intervals corresponding to an alpha level of half the value specified. For example, if the α level specified in the dialog was 0.05, Prism would incorrectly use an α value of 0.025 (corresponding to a 97.5% confidence level).

Recall that larger percentage confidence intervals are always wider than smaller percentage confidence intervals for the same data. In other words, a 97.5% confidence interval calculated for a given set of data will always be wider than a 95% confidence interval calculated for the same data. Because Prism reported the confidence interval corresponding to α/2 instead of α, the reported confidence intervals were always wider (more conservative) than they should have been. However, the P values were always reported correctly. The result is that in some cases the reported P value and the confidence interval reported by Prism may not have “agreed”. If the P value that Prism reported (correctly) was between 0.025 and 0.05, the 95% confidence interval that Prism reported (incorrectly) would have included zero in its range.

Obtaining correct 95% confidence intervals in Prism versions prior to 9.3.1

Because Prism versions prior to 9.3.1 incorrectly reported a confidence interval corresponding to α/2 instead of α, it is possible to get the correct values for a 95% confidence interval in these versions by understanding the relationship between alpha and the confidence level.

In the Options tab of the “Parameters: One-Way ANOVA (and Nonparametric or Mixed)” analysis dialog, there is a dropdown menu for the “Family-wise alpha threshold and confidence level” within the “Multiple comparisons options” section. In this dropdown menu, select “0.10 (90% confidence interval)”. Prism will (incorrectly) calculate a confidence interval corresponding to α/2, or - in this case - 0.10/2 = 0.05. In this case, the confidence interval that Prism reports will be labeled as a 90% confidence interval, but will actually report the correct values for a 95% confidence interval.