The length of the receiver operating characteristic curve and the two cutoff Youden index within a robust framework for discovery, evaluation, and cutoff estimation in biomarker studies involving improper receiver operating characteristic curves

Abstract

During the early stage of biomarker discovery, high throughput technologies allow for simultaneous input of thousands of biomarkers that attempt to discriminate between healthy and diseased subjects. In such cases, proper ranking of biomarkers is highly important. Common measures, such as the area under the receiver operating characteristic (ROC) curve (AUC), as well as affordable sensitivity and specificity levels, are often taken into consideration. Strictly speaking, such measures are appropriate under a stochastic ordering assumption, which implies, without loss of generality, that higher measurements are more indicative for the disease. Such an assumption is not always plausible and may lead to rejection of extremely useful biomarkers at this early discovery stage. We explore the length of a smooth ROC curve as a measure for biomarker ranking, which is not subject to directionality. We show that the length corresponds to a $\varphi$ divergence, is identical to the corresponding length of the optimal (likelihood ratio) ROC curve, and is an appropriate measure for ranking biomarkers. We explore the relationship between the length measure and the AUC of the optimal ROC curve. We then provide a complete framework for the evaluation of a biomarker in terms of sensitivity and specificity through a proposed ROC analogue for use in improper settings. In the absence of any clinical insight regarding the appropriate cutoffs, we estimate the sensitivity and specificity under a two-cutoff extension of the Youden index and we further take into account the implied costs. We apply our approaches on two biomarker studies that relate to pancreatic and esophageal cancer.

Keywords: $\varphi$ divergence; Youden; isoperimetric; kernels; likelihood ratio; optimal ROC; sensitivity; specificity; stochastic ordering; two-cutoff ROC.

Figure 1.

Upper panels: An example of a perfect biomarker where the densities of the controls and the diseased (left) comply with a traditional framework, under which the stochastic ordering X < Y holds, and the corresponding ROC curve is in the right panel. Corresponding metrics: AUC = 1, pAUC(t₁, t₂; t₂ > t₁) = t₂ − t₁, area between the ROC and the reference diagonal= 0.5, proposed length= 2. Lower panels: An example of a perfect biomarker in which the stochastic ordering X < Y cannot be assumed. Corresponding metrics: AUC = 0.5, pAUC(t₁, t₂;t₂ > t₁) = 0.5(t₂ − t₁), area between the ROC and the reference diagonal= 0.25, proposed length = 2 (the proposed length is immune to directionality and characterizes the underlying biomarker as perfect in both cases, as opposed to all other measures.)

Figure 2.

Geometric representation of the problem of finding the max and min area under the curve, when assuming convexity for a given length of the rope (ROC). Under convexity, it is enough to study the area between the reference diagonal and the rope. Hence, we only can focus on the right panel (b), which is equivalent to the left panel (a).

Figure 3.. Upper panels:

Geometric representation of the maximum area for a given length when the length is larger than $\frac{\pi }{2}$ (panel (a)), and the minimum area for the same given length (panels (b) and (c)). Both panels (b) and (c) correspond to the same area. All three panels correspond to the same length. Lower panels: Geometric representation of the maximum area for a given length when the length is smaller than $\frac{\pi }{2}$ (panel (a)), and the minimum area for the same given length (panels (b) and (c)). Both panels (b) and (c) correspond to the same area. All three panels correspond to the same length.

Figure 4.

Example of a sensitivity surface when the density of the controls is N(8,1) and the density of the cases is a two component normal mixture: 0.5N(6, 1) + 0.5N(10, 1). The sensitivity plot is given under four different angles (panels above) for better visualization. We observe that when c₁ is very small and c₂ is very large then the sensitivity yields very low values since the mass of both densities is low below c₁ and beyond c₂. In addition, if c₁ is very close to c₂ then we expect that almost all individuals that are diseased will be categorized as such since it is very likely that they will not be between c₁ and c₂ and thus the sensitivity for those cutoffs is very high (red regions).

Figure 5.

Example of a specificity surface when the density of the controls is N(8, 1) and the density of the cases is a two component normal mixture: 0.5N(6, 1) + 0.5N(10, 1). The specificity plot is given under four different angles (panels above) for better visualization. We observe that when c₁ is very small and c₂ is very large then the specificity yields very high values as expected since most healthy individuals will lie within (c₁, c₂). In addition, if c₁ is very close to c₂ then we expect that almost all individuals that are healthy will be categorized as diseased and since it is very likely that they will not be between c₁ and c₂ and thus the specificity for those cutoffs is very low (blue regions).

Figure 6.

ROC curves for the hypothetical example with markers M₁ and M₂. Both ROC curves are illustrated and would tempt us to consider M₁ as an overall better biomarker. The gROC of marker 2 is presented as well showing its potential in actually outperforming M₁. The gROC of M₂ and the ROC of M₂ have exactly the same length that is larger than the length of M₁.

Figure 7.

Scatterplot of the AUCs and proposed lengths of all 121 autoantibodies.

Figure 8.

Left Panel: Traditional ROC curves for the top performing autoantibody: dashed line refers to the empirical, and solid line refers to the kernel-smoothed based one. The corresponding AUCs and 95% CIs are 0.4468(0.3392, 0.5544) and 0.3892(0.3055,0.4730), respectively. The estimated kernel-based length with the corresponding 95% CI is 1.8255 (1.7846–1.8748). Right panel: The corresponding kernel-based estimated gROC. The area under this kernel based estimate of gROC is 0.9154.

Figure 9.

Top left: Egg shaped 95% confidence region for the sensitivity and specificity at the Youden-based estimated cutoffs. Top right: Underlying kernel densities of the healthy and the diseased. Middle: Surfaces of sensitivity and specificity along with the estimated Youden-based optimal pair of cutoffs. Bottom left: Contour plot of the surface of sensitivity for all possible cutoffs, along with the estimated Youden-based optimal pair of cutoffs (black dot). Bottom right: Contour plot of the surface of specificity for all possible cutoffs, along with the estimated Youden-based optimal pair of cutoffs (black dot).

Figure 10.

ESCC data (probe 207039).Top left: Egg shaped 95% confidence region for the sensitivity and specificity at the Youden-based estimated cutoffs. Top right: Underlying kernel densities of the healthy and the diseased. Middle: Surfaces of sensitivity and specificity along with the estimated Youden-based optimal pair of cutoffs. Bottom left: Contour plot of the surface of sensitivity for all possible cutoffs, along with the estimated Youden-based optimal pair of cutoffs (black dot). Bottom right: Contour plot of the surface of specificity for all possible cutoffs, along with the estimated Youden-based optimal pair of cutoffs (black dot).

Figure 11.

ESCC data (probe 209644). Top left: Egg shaped 95% confidence region for the sensitivity and specificity at the Youden-based estimated cutoffs. Top right: Underlying kernel densities of the healthy and the diseased. Middle: Surfaces of sensitivity and specificity along with the estimated Youden-based optimal pair of cutoffs. Bottom left: Contour plot of the surface of sensitivity for all possible cutoffs, along with the estimated Youden-based optimal pair of cutoffs (black dot). Bottom right: Contour plot of the surface of specificity for all possible cutoffs, along with the estimated Youden-based optimal pair of cutoffs (black dot).

Figure 12.

gROCs for both probes discussed in the application that refers to the esophageal data. Left panel: gROC for probe 209644 with an area under it equal to 0.8238. The two cutoff Youden-based sensitivity and specificity are derived to be (0.8754, 0.6659). The area under the usual empirical ROC estimate is 0.5421 (0.4175–0.6667) which implies that this marker would have been discarded as uninformative in spite of its discriminatory ability. Right panel: gROC for probe 207039 with an area under it equal to 0.8858. The two cutoff Youden-based sensitivity and specificity are derived to be (0.9180, 0.7648). The area under the usual empirical ROC estimate is 0.5185 (0.3963–0.6397) which implies that this marker would have been discarded as uninformative in spite of its discriminatory ability.