How is the Dunn's post test calculated following the Kruskal-Wallis test (nonparametric comparison of three or more groups).

InStat and Prism use a standard method. One source is Daniel's Applied nonparametric statistics, second edition page 240-241. The original reference is . O.J. Dunn, Technometrics, 5:241-252, 1964.

First compute the value of alpha that accounts for multiple comparisons. Divide the usual significance threshold, 0.05, that will apply to the entire family of comparisons, by the number of comparisons you are making. If you compare each group with every other group, you are making K*(K-1)/2 comparisons, where K is the number of groups. Then find the value of z from the normal distribution that corresponds to that two-tailed probability. This free calculator will help. For example, if there are 4 groups, you are making 6 comparisons, and the critical value of z (using the usual 0.05 significance level for the entire family of comparions) is the z ratio that corresponds to a probability of 0.05/6 or 0.008333. That z ratio is  2.638.

To compare group i and j, find the absolute value of the difference between the mean ranks in group i and the mean ranks in group j. If there are no ties, divide this difference in mean ranks by the square root of [(N*(N+1)/12)*(1/Ni + 1/Nj]. Here N is the total number of data points in all groups, and Ni and Nj are the number of data points in the two groups being compared.  If there are ties, divide this difference in mean ranks by the square root of [(N*(N+1) - Sum(Ti^3 - Ti) / (N - 1)) / 12 * (Ni + 1/Nj), where Ti is the number of ties in the i-th group of ties. If the ratio calculated in the preceding paragraph is larger that the critical value of z computed in the paragraph before that, then conclude that the difference is statistically significant.

If there are ties (two values are the same), those values will share the same rank. For example if two values tie for ranks 3 and 4, they both get rank 3.5 and that is the rank used in computing the mean rank mentioned above.

Note that when comparing two groups, this method computes the mean rank of each group from the overall analysis. You don't rank just the values in those two groups. Rank all the values in all the groups, then cmpare the mean ranks of the two groups being compared.