The term 'outlier' is defined fairly vaguely, but refers to a value that is far from the others. In Prism's nonlinear regression, an outlier is a point that is far from the best-fit curve defined by robust regression.
Of course, there is some possibility that an outlier really comes from the same Gaussian population as the others, and just happens to be very high or low. You can set the value of Q to control how aggressively Prism defines outliers.
Nonlinear regression is usually used with experimental data, where X is a variable like time or concentration or some other variable you manipulate in the experiment. Since all the scatter is due to experimental error, it can make sense to eliminate any extreme outlier since it is almost certainly the result of an experimental mistake.
In other situations, each data point can represent a different individual. In this case, an outlier may not be due to experimental mistakes, but rather be the result of biological variation, or differences in some other variable that is not included in your model. Here, the presence of the outlier may be the most interesting finding in the study. While the ROUT outlier method might prove useful to flag an outlier in this situation, it would be a big mistake to automatically exclude such outliers without further thought (or experimentation).
In quality control analyses, an outlier can tell you about a process that is out of control. You wouldn't want to delete outliers, without first figuring out why the value is far from the others. The outlier might be telling you something important.