For a Mann Whitney U test, how does Prism handle ties?

The Mann-Whitney compare two groups of values without making any assumption about the distribution the values were sampled from. It does this by ranking all the values from low to  high and then comparing the average rank of the two groups. It reports a P value testing the null hypothesis that the two populations have identical distributions, so any discrepancy in mean rank is just a matter of random sampling.

If two values are identical, they tie for the same rank. This situation complicates calculation of a  P value. 

The test was developed to analyze data from continuous measurements like enzyme activity or weight or blood pressure. In this situation, ties will be rare. How rare? It depends on how many digits of precision are recorded.  

Some people use the Mann-Whitney test to compare categorical outcomes, with the categories coded as numbers.  In this situation, there are only a few possible values (the number of categories) so there will be lots of tied ranks. I am dubious about how useful the Mann-Whitney test is in this situation. Rather than do the Mann-Whitney test, consider analyzing the data as categorical data using a chi-square test to compare the two groups.

The test statistic for the Mann-Whitney test is the U statistic. Prism uses a standard method to correct for ties when it computes U, and the sum of signed ranks. Computing a P value from U (or from the sum of signed ranks; the two end up being equivalent) is not so straightforward. Geyer suggests computing a range of P values, which he calls fuzzy P values. However, this approach is not widely utilized, and is not the technique that Prism uses to calculate P values. Instead, Prism uses a technique which calculates an exact P value even in the presence of ties. To do this, Prism first calculates the U statistic for the observed data. Then, using the ranks of all of the data, Prism determines how many different permutations of the data would result in a U statistic equal to or less than the U statistic calculated for the observed data. This number is divided by the total number of possible permutations of the data, yielding a fraction of total U statistics that are equal to or less than the U statistic calculated for the observed data. The exact P value is equal to this fraction.

The exception to this approach is when there are more than 100 values in the smaller of the two data sets being compared. In this situation, Prism will always calculate an approximate P value (regardless of ties). This is accomplished by first converting the calculated U statistic for the observed data into a Z statistic (assuming a Gaussian distribution). This Z statistic is then used to determine the corresponding P value.

Earlier versions of Prism used different techniques to calculate P values:

• Prism 5 computes an exact exact P value when the samples are small and when there are no ties. When there are ties, Prism 5 always uses the approximate method (even with small samples), which converts U or sum-of-ranks to a Z value. It then looks up that value on a Gaussian distribution to get a P value.
• Prism 3 and 4 used suboptimal algorithms for computing a P value when there are ties. Prism 5.00 introduced an improved algorithm, which was fixed in 5.02 and 5.0b (prior to that, Prism correctly accounted for tied values within the first column, and ties between the first and second column, but did not correctly account for ties within the second column). This fix is also in InStat 3.1.  If you use earlier releases  of Prism or InStat, please update, especially if your data has lots of ties.