Improvement of risk prediction by genomic profiling: reclassification measures versus the area under the receiver operating characteristic curve

Abstract

Reclassification is observed even when there is no or minimal improvement in the area under the receiver operating characteristic curve (AUC), and it is unclear whether it indicates improved clinical utility. The authors investigated total reclassification, net reclassification improvement, and integrated discrimination improvement for different DeltaAUC using empirical and simulated data. Empirical analyses compared prediction of type 2 diabetes risk based on age, sex, and body mass index with prediction updated with 18 established genetic risk factors. Simulated data were used to investigate measures of reclassification against DeltaAUCs of 0.005, 0.05, and 0.10. Total reclassification and net reclassification improvement were calculated for all possible cutoff values. The AUC of type 2 diabetes risk prediction improved from 0.63 to 0.66 when 18 polymorphisms were added, whereas total reclassification ranged from 0% to 22.5% depending on the cutoff value chosen. In the simulation study, total reclassification, net reclassification improvement, and integrated discrimination improvement increased with higher DeltaAUC. When DeltaAUC was low (0.005), net reclassification improvement values were close to zero, integrated discrimination improvement was 0.08% (P > 0.05), but total reclassification ranged from 0 to 6.7%. Reclassification increases with increasing AUC but predominantly varies with the cutoff values chosen. Reclassification observed in the absence of AUC increase is unlikely to improve clinical utility.