Fixed bugs: Incorrect P values from Fisher's Exact test or chi-square test

Bug 1: In Prism 5, and  6.00, 6.01, 6.0a and 6.0b, the one-sided P value is sometimes reported as "undefined".

This only happens when you request a one-tail P value and if (A div B) = 1 and (C div D) = 1, or (A div C) = 1 and (B div D) = 1, where div is the operation of integer division (e.g. 9 div 6 = 1). Will fix in 6.02 and 6.0c.  In most cases, this happens when the P value would be large, but you cannot rely on this. The P value might be small. If the P value is reported, it is correct. The problem is that rarely, Prism reports "undefined" instead of a P value. 

Bug 2: With huge numbers, P value can be wrong (fixed in 6.00 and 6.0a)

You enter four values, which we label like this:

A  B
C  D

If the product of B times C is huge, Prism will encounter a math error and report an incorrect value for chi-square and P, but no error message. This only happens when the product of B times C exceeds 2,147,483,648. With such large numbers, Prism won't do a Fisher's test even if you ask for it, but does the chi-square instead. This bug is in Prism 3-5, but fixed in 6.00 and 6.0a.

You can bypass this bug, in most cases, by swapping the two columns. Only if the product of A times D is also huge will this strategy not work.

Bug 3:  With large samples, P value is always two-tailed  (fixed in 5.03 and 5.0c)

 

When the sample sizes are large, the exact Fisher calculations would be slow, so Prism switches to perform the chi-square test instead. When this happens, Prism (up to 5.03 and 5.0c) always reported a two-tail P value, even if you requested a one-tail P value. Fixed in 5.04 and 5.0d.

Bug 4: P too low rarely in 5.02 and 5.0b (fixed in 5.03 and 5.0c)

Prism 5.02 (Windows) and 5.0b (Mac) included a fix to a trivial bug in Fisher's exact test (when the two groups are identical, the P value should be 1.00 but earlier versions of Prism sometimes reported P values slightly greater than 1.0).  Unfortunately, that fix introduced a new bug described here:

The bug occurs only when:

• You are using Prism 5.02 (Windows) or 5.0b (Mac). Earlier versions did not have this bug. Neither does InStat 3.0 or 3.1. 
• You have entered a symmetrical contingency table. A table is symmetrical when either the two row totals are identical, or the two column totals are identical. 
• You have chosen a two-tail (two-sided) P value. One-tail P values are computed correctly. 
• The sum of the four values entered into the contingency table is less than 2001. When the sum is greater than 2000, Prism performs chi-square test even if you asked for Fishers, because the Fisher computations would be so slow. 

The result of the bug is that the reported P value will be too low in some, but not all cases. In many cases, the discrepancy is tiny and won't affect your conclusions. In other cases, the discrepancy is larger and may affect your conclusion.

The problem occurs in about half the cases when the values are large (the sum of the four values exceeds 100). When the values are smaller, the problem occurs less often, but still can happen. It depends on the values entered. 

We have fixed both of these bugs with Prism: 5.03 and 5.0c.

Workaround

With symmetrical contingency tables, the two-tail P value is exactly twice the one-tail P value (that is not always true with contingency tables that are not symmetrical). Therefore, to bypass the bug, ask Prism to compute a one-tail (one-sided) P value. This is a choice in the Parameters dialog for analyzing contingency table. To compute a two-tail P value, simply double the one-tail P value. 

Example

 

	Outcome A	Outcome B	Total
Group 1	98	89	187
Group 2	20	29	49
Total	118	118	236

This table is symmetrical because the two column totals are identical (118 each). The table would also be symmetrical if the row totals were equal. Of course, you only enter four values into the contingency data table of Prism. You don't enter the column or row totals, which Prism computes. 

Prism 5.02 and 5.0b report that the two-tail P value is 0.1534. This is incorrect. To obtain the correct two-tail P value, ask Prism to compute the one-tail P value. The result is 0.0994. Manually double that value to obtain the correct two-tail P value, which is 0.1988.

Mathematical details

Computing a one-tail P value from a Fisher's test is straightforward, but there are three different algorithms for computing the two-tail P value. Prism (and also InStat, and the QuickCalc since 5-April-2004) compute the two-sided P value using the method of summing small P values. Most statisticians recommend this approach, but some programs use a different approach. 

The calculation works like this:

1. Create every possible contingency table with the same row and column totals as in the actual table.
2. For each table, compute the probability that these results would have occurred under the null hypothesis. These probabilities will sum to 1.00. 
3. Total all those probabilities that are less than or equal to the probability of the actual table. That sum is the two-tail P value. 

The third step is the problem. With a symmetrical contingency table, there is always another table that has exactly the same probability as the original, and its probability should be included in the total (step 3 above) that becomes the two-tail P value. However the calculations that generate that probability are done in a different order for the two equivalent tables, and computer round off issues can result in a probabilities that are not quite equal. If the other table ends up with a slightly higher probability than the original table, it is not included in the sum of probabilities of step 3, so Prism reports a two-tail P value that is smaller than it should be. The fundamental problem is that asking a computer whether A equals B is not reliable with floating point (fractional) values, because of these round off errors. 

Prism 5.03 and 5.0c fix the bug by computing the two-tail P values for symmetrical tables differently than it computes asymmetrical tables. For symmetrical tables, Prism will compute the one-tail P value and double it (with special code to deal with a P value equal to 1.0).