On contemporaneous controls, unlikely outcomes, boxes and replacing the 'Student': good statistical practice in pharmacology, problem 3

Abstract

This paper is intended to assist pharmacologists to make the most of statistical analysis and avoid common errors. A scenario, in which an experimenter performed an experiment in two separate stages, combined the control groups for analysis and found some surprising results, is presented. The consequences of combined controls are discussed, appropriate display and analysis of the data are described, and an analysis of the likelihood of erroneous conclusions is made. Comparisons between data from separately conducted experimental series are hazardous when there is any possibility that the properties of the experimental units have changed between the series. Experiments that have been performed independently should be analyzed independently. Unlikely or surprising results should be treated with caution and a high standard of evidence should be required, and verification by repeated experiments should be performed and reported. Box and whisker plots contain more information than plots more commonly used to display for qualitative variables and should be used where the sample size is large enough (say, n > or = 5). In most biomedical experiments the observations are not random samples from large populations as assumed by conventional parametric analyses such as Student's t-test, and so permutation tests, which do not lose their validity when a sampled population is non-normal or when the data are not random samples, should frequently be used instead of Student's t-tests.

Figure 1

Results of the experiments described in this problem. The indicated P-values are for comparisons between the test groups and control. Error bars are s.e.

Figure 2

Results of the experiments of this problem replotted to make the experimental design and the outcomes clear. (a) graph showing individual data points (n=10 for each treatment) and their means. (b) the same data shown as a box and whisker plot. In this plot the whiskers extend to the largest and smallest data points, the box extends from the upper quartile to the lower quartile and is crossed by a line at the median of the data.

Figure 3

Tree diagrams that illustrate the probabilities of obtaining the four possible combinations of null hypothesis (H₀) condition (true or false) and experimental outcome (sufficient or insufficient evidence against H₀). (a) generic tree with conventional symbols for the probability of a false-positive outcome (α) and a false-negative outcome (β). (b) tree with the probabilities of false-positive and false-negative outcomes as specified in the text. (c) tree for circumstances in which a true effect (false H₀) is common (9 times out of 10). (d) tree for circumstances in which a true effect (false H₀) is rare (1 time out of 10).