Statistical Issues in TBI Clinical Studies

Abstract

The identification and longitudinal assessment of traumatic brain injury presents several challenges. Because these injuries can have subtle effects, efforts to find quantitative physiological measures that can be used to characterize traumatic brain injury are receiving increased attention. The results of this research must be considered with care. Six reasons for cautious assessment are outlined in this paper. None of the issues raised here are new. They are standard elements in the technical literature that describes the mathematical analysis of clinical data. The purpose of this paper is to draw attention to these issues because they need to be considered when clinicians evaluate the usefulness of this research. In some instances these points are demonstrated by simulation studies of diagnostic processes. We take as an additional objective the explicit presentation of the mathematical methods used to reach these conclusions. This material is in the appendices. The following points are made: (1) A statistically significant separation of a clinical population from a control population does not ensure a successful diagnostic procedure. (2) Adding more variables to a diagnostic discrimination can, in some instances, actually reduce classification accuracy. (3) A high sensitivity and specificity in a TBI versus control population classification does not ensure diagnostic successes when the method is applied in a more general neuropsychiatric population. (4) Evaluation of treatment effectiveness must recognize that high variability is a pronounced characteristic of an injured central nervous system and that results can be confounded by either disease progression or spontaneous recovery. A large pre-treatment versus post-treatment effect size does not, of itself, establish a successful treatment. (5) A procedure for discriminating between treatment responders and non-responders requires, minimally, a two phase investigation. This procedure must include a mechanism to discriminate between treatment responders, placebo responders, and spontaneous recovery. (6) A search for prodromes of neuropsychiatric disorders following traumatic brain injury can be implemented with these procedures.

Keywords: Mahalanobis distance; neuropsychiatric diagnosis; research design; statistical errors; statistical variability; treatment effects.

Figure 1

P_SAME ≠ P_ERROR. Two normal distributions: μ_A = 3.2117, σ_A = 14.8328 (in blue), μ_B = − 3.1433, (in red) σ_B = 14.8255, N_A = N_B = 500. Given assumptions that the distributions are normal and that an optimal Bayesian classifier is used to classify individual elements, P_SAME = 2.1096 × 10⁻¹¹ and P_{ERROR − FORMULA} = 0.4078.

Figure 2

Sensitivity of discrimination and backward elimination: The between-group Mahalanobis distance D_A,B, the coefficient of determination R_A,B, and the theoretical probability of error in a pairwise classification, P_{ERROR − FORMULA} are plotted as a function of the number of measures eliminated from the discrimination. At each step the least significant variable was removed. From Watanabe et al. (9).

Figure 3

Error rate in a k-fold cross validation as a function of the number of variables eliminated from the classifier. The elimination sequence in the upper trace (denoted by circles) was determined by a backward elimination. The elimination sequence of the lower trace (triangles) was determined in a sequential correlation deletion. From Watanabe et al. (9).

Figure 4

P_SAME (G_A, G_B) as a function of effect size. P_SAME (G_A, G_B) was calculated as a function of effect size (equivalently the one-dimensional Mahalanobis distance) for different group sizes. In all calculations, the number of members in each group was the same, N_A = N_B. The populations are N_A = N_B = 10 (top curve), 20, 50, 100, 200, 500 (bottom curve).