In most experimental situations, your goal is to show that one treatment is better than another. But in some situations, your goal is just the opposite -- to prove that a one treatment is indistinguishable from another. You are not trying to prove that one treatment makes a statistically significant difference in the outcome. Rather, you are trying to prove that two treatments are essentially equivalent, that any difference is of no practical consequence.

It is tempting to simply run a standard statistical test (i.e. t test if the outcome is on a continuous scale, Fisher's test if the outcome is binary) to see if the result is statistically significant, and then interpret the results simply. If the difference is statistically significant, then it seems clear that the two treatments are not equivalent. And if the difference is not statistically significant, it seems to make sense to conclude that the two treatments are equivalent.

But that approach is wrong, and leads to invalid conclusions. It isn't difficult to correctly interpret experiments testing for equivalence. Look at the confidence intervals, use plenty of common sense, and don't bother with P values or statements of statistical significance. I've writen a short article that explains how to do it. I've also created a new page in the GraphPad resource library, with links to other articles (and a book) about equivalence analysis.

John Hartung, of SUNY HSC Brooklyn, sent me an interesting analogy to help teach the concept of statistical power.

You send your child into the basement to find a tool. He comes back and says "it isn't there". What do you conclude? Well, it depends....