Net Reclassification Index Statistics Do Not Help Assess New Risk Models

Abstract

When evaluating a new risk factor for disease (eg, a measurement from imaging studies), many investigators examine its value above and beyond existing biomarkers and risk factors. They compare the performance of an "old" risk model using established predictors and a "new" risk model that adds the new factor. Net reclassification index (NRI) statistics are a family of metrics for comparing two risk models. NRI statistics became popular in some medical fields and have appeared in high-impact journals. This article reviews NRI statistics and describes several issues with them. Problems include unacceptable statistical behavior, incorrect statistical inferences, and lack of interpretability. NRI statistics are unhelpful (at best) and misleading (at worst).

Graphical abstract

Figure 1:

Mathematical definition of the net reclassification index (NRI). The NRI seeks to compare a new risk model to an existing model. “Event” and “nonevent” refer to patients who would and would not go on to experience an event without intervention. “Up” refers to patients who move to a higher risk category with the new risk model. “Down” refers to patients who move to a lower risk category. The event NRI and nonevent NRI are defined as shown in the equation. The (overall) NRI is the sum of the event NRI and nonevent NRI. Note that the NRI combines four proportions but is not itself a proportion. The quantitative value of an NRI statistic can range from −2 to 2. P = probability.

Figure 2:

Three-category net reclassification index (NRI) for a sample of 1000 patients from a population with 90% events (patients who would go on to experience an event without intervention) and 10% nonevents (patients who would not go on to experience an event without intervention). Overall NRI is higher in example 2 compared with example 1. Yet, in example 2 more patients have worse risk classification (orange cells) compared with better risk classification (green cells) with a switch from the old to the new risk model. In other words, more patients would be harmed than helped by switching from the old risk model to the new risk model, even though the NRI statistic is positive. This behavior arises because NRI ignores the relative proportions of events and nonevents in the population. In example 3, the percentage of event patients reclassified and the percentage of nonevent patients reclassified are identical to those in example 1; the NRI statistics are also identical to those in example 1. However, compared with example 1, the new risk model in example 3 reclassifies more event patients from high risk to low risk instead of from high risk to medium risk and reclassifies more nonevent patients from low risk to high risk instead of from low risk to medium risk. The NRI statistics are identical for example 1 and example 3 because the NRI only accounts for the direction of changes and not their magnitude; thus, the NRI ignores crucial information about the clinical impact of reclassification.