Probabilistic index: an intuitive non-parametric approach to measuring the size of treatment effects

Abstract

Effect sizes (ES) tell the magnitude of the difference between treatments and, ideally, should tell clinicians how likely their patients will benefit from the treatment. Currently used ES are expressed in statistical rather than in clinically useful terms and may not give clinicians the appropriate information. We restrict our discussion to studies with two groups: one with n patients receiving a new treatment and the other with m patients receiving the usual or no treatment. The standardized mean difference (e.g. Cohen's d) is a well-known index for continuous outcomes. There is some intuitive value to d, but measuring improvement in standard deviations (SD) is a statistical concept that may not help a clinician. How much improvement is a half SD? A more intuitive and simple-to-calculate ES is the probability that the response of a patient given the new treatment (X) is better than the one for a randomly chosen patient given the old or no treatment (Y) (i.e. P(X > Y), larger values meaning better outcomes). This probability has an immediate identity with the area under the curve (AUC) measure in procedures for receiver operator characteristic (ROC) curve comparing responses to two treatments. It also can be easily calculated from the Mann-Whitney U, Wilcoxon, or Kendall tau statistics. We describe the characteristics of an ideal ES. We propose P(X > Y) as an alternative index, summarize its correspondence with well-known non-parametric statistics, compare it to the standardized mean difference index, and illustrate with clinical data.