False Discovery Rates in PET and CT Studies with Texture Features: A Systematic Review

Abstract

Purpose: A number of recent publications have proposed that a family of image-derived indices, called texture features, can predict clinical outcome in patients with cancer. However, the investigation of multiple indices on a single data set can lead to significant inflation of type-I errors. We report a systematic review of the type-I error inflation in such studies and review the evidence regarding associations between patient outcome and texture features derived from positron emission tomography (PET) or computed tomography (CT) images.

Methods: For study identification PubMed and Scopus were searched (1/2000-9/2013) using combinations of the keywords texture, prognostic, predictive and cancer. Studies were divided into three categories according to the sources of the type-I error inflation and the use or not of an independent validation dataset. For each study, the true type-I error probability and the adjusted level of significance were estimated using the optimum cut-off approach correction, and the Benjamini-Hochberg method. To demonstrate explicitly the variable selection bias in these studies, we re-analyzed data from one of the published studies, but using 100 random variables substituted for the original image-derived indices. The significance of the random variables as potential predictors of outcome was examined using the analysis methods used in the identified studies.

Results: Fifteen studies were identified. After applying appropriate statistical corrections, an average type-I error probability of 76% (range: 34-99%) was estimated with the majority of published results not reaching statistical significance. Only 3/15 studies used a validation dataset. For the 100 random variables examined, 10% proved to be significant predictors of survival when subjected to ROC and multiple hypothesis testing analysis.

Conclusions: We found insufficient evidence to support a relationship between PET or CT texture features and patient survival. Further fit for purpose validation of these image-derived biomarkers should be supported by appropriate biological and statistical evidence before their association with patient outcome is investigated in prospective studies.

Fig 1. Study flow diagram according to PRISMA guidelines.

Fig 2. Probability of a false positive result based on number of hypotheses tested per study (blue columns) for all study categories.

5% type-I error probability = red line, average type-I error probability (76%) over all studies = green line (Note—additional inflation of the type-I error probability due to the use of the optimum cut-off approach is not included here).

Fig 3. Studies from categories A and B after adjustments for optimum cut-off approach and/or multiple hypotheses testing.

Green column demonstrates the smallest published p-value per study, the red the P_cor for the optimum cut-off approach, and the blue the corrected statistical significance level based on Hochberg-Benjamini method. For a study to have a statistical significant result the red column value should be smaller than the green blue which is not the case for any of them. For study [19] the green and red column are identical as investigators did not use the optimum cut-off approach. Studies [31,33] and [41] were excluded as they did not provide a summary of their p-values for correction, and had adjusted the results for multiple hypotheses, respectively.

Fig 4. Area under the curve (AUC) values from receiver operating characteristic (ROC) analysis of 100 random variables.

The variables are ordered by decreasing AUC values.

Fig 5. Statistical significance of Kaplan-Meier analysis for 100 random variables using the optimum cut-off approach.

The variables are ordered by increasing p-values. Overall 10% of the random variables are statistically significant predictors of survival.

Fig 6. Kaplan Meier curves based on optimum cut-off value for the random variable 1.

Fig 7. Kaplan Meier curves based on mean value for the random variable 1.