Estimation and inference on the partial volume under the receiver operating characteristic surface

Abstract

measures of biomarker accuracy that employ the receiver operating characteristic surface have been proposed for biomarkers that classify patients into one of three groups: healthy, benign, or aggressive disease. The volume under the receiver operating characteristic surface summarizes the overall discriminatory ability of a biomarker in such configurations, but includes cutoffs associated with clinically irrelevant true classification rates. Due to the lethal nature of pancreatic cancer, cutoffs associated with a low true classification rate for identifying patients with pancreatic cancer may be undesirable and not appropriate for use in a clinical setting. In this project, we study the properties of a more focused criterion, the partial volume under the receiver operating characteristic surface, that summarizes the diagnostic accuracy of a marker in the three-class setting for regions restricted to only those of clinical interest. We propose methods for estimation and inference on the partial volume under the receiver operating characteristic surface under parametric and non-parametric frameworks and apply these methods to the evaluation of potential biomarkers for the diagnosis of pancreatic cancer.

Keywords: Biomarkers; classification; delta Method; kernels; receiver operating characteristic surface.

Figure 1.

For a single hypothetical marker this illustrates three (out of many) different partial volumes one could consider depending on the clinical interest. Plots a to c show the pVUS with the ROC surface for the biomarker of interest. Plot a shows the pVUS after forcing a true class rate for group I of 0.4–0.6 and for group 3 a range of 0.1–0.3. The corresponding pVUS is visualized by the purple tube. Analogously, in plot b, we force specific ranges for $T{C}_3$ (group 3) and $T{C}_2$ (group 2), and in plot c, we force specific ranges for $T{C}_1$ and $T{C}_2$. Plots d to f display the same pVUS regions for the same biomarker as plots a to c, respectively, but with the uninformative ROC surface for comparison. pVUS: partial volume under the receiver operating characteristic surface; ROC: receiver operating characteristic; TC: true classification.

Figure 2.

The ROC surface for two hypothetical markers and their corresponding feasible regions. Plot a shows the uninformative ROC surface and the ROC surface for hypothetical marker in which the ROC surface does not intersect the uninformative ROC surface, and plots b to d show the marker’s feasible regions for $pVU{S}_{T{C}_1,T{C}_3},pVU{S}_{T{C}_2,T{C}_3}$, and $pVU{S}_{T{C}_1,T{C}_2}$. Plot e shows the uninformative ROC surface and the ROC surface for hypothetical marker in which the ROC surface intersects the uninformative ROC surface, and plots f to h show the marker’s feasible regions for $pVU{S}_{T{C}_1,T{C}_3},pVU{S}_{T{C}_2,T{C}_3}$, and $pVU{S}_{T{C}_1,T{C}_2}$. These plots illustrate the following areas on each plane: A, which consists of all points that fall in the projection of the ROC surface of interest on the given plane but not in the projection of the uninformative surface; B, which consists of all points that fall in the projection of both the uninformative surface and the ROC surface of interest and; C, which consists of all points that fall in the projection of the uninformative surface onto the given plane, but not in the projection of the ROC surface of interest. pVUS: partial volume under the receiver operating characteristic surface; ROC: receiver operating characteristic; TC: true classification.

Figure 3.

Plots of the estimated coverage of the pVUS CI’s under scenarios involving normal, log-normal, and gamma distributed biomarker scores at various levels of the VUS and pVUS. The colors denote the true VUS and the sizes of the dots denote the true pVUS. The dotted line represents a coverage of 95%. For corresponding scaled pVUS see Supplemental Appendix E. pVUS: partial volume under the receiver operating characteristic surface; CI: confidence interval; VUS: volume under the receiver operating characteristic surface.

Figure 4.

Plots of the estimated size (at 5% significance) and power of the statistical test of the pVUS involving normal, log-normal, and gamma distributed biomarker scores at various levels of $pVU{S}_{SC}$ for a restricted region of $T{C}_3\in [0.1,0.3]$ and $T{C}_1\in [0.1,0.3]$. The pVUS values corresponding to the $pVU{S}_{SC}$ values shown on the x-axis are all taken on the same region of interest. Furthermore, for the selected region of interest in these simulations, an uninformative marker exhibits a pVUS of 0.024. Thus, the size of the test was evaluated at $pVUS=0.024$, and the power of the test was evaluated at various values of $pVUS>0.024$. The dotted lines represent powers of 5%, 80%, and 90%. We note that the corresponding $pVU{S}_{SC}$ values in these simulations range from 0.167 to 0.717. pVUS: partial volume under the receiver operating characteristic surface; TC: true classification.

Figure 5.

The estimated feasible regions and feasible regions for inference of the seven pancreatic ductal adenocarcinoma (PDAC) biomarkers. The feasible regions are represented by the unions of regions $\mathrm{A}$ and $\mathrm{B}$ $(A\cup B)$ and region $\mathrm{B}$ is the feasible region for inference.

Figure 6.

The estimates and 95% Cl’s for the VUS, scaled pVUS, and pVUS for seven potential PDAC biomarkers. The pVUS is defined by $T{C}_1$ boundaries $p_1=0,p_2=0.2$, and $T{C}_3$ boundaries, $q_1=0.6,q_2=0.8$.

Figure 7.

The estimated distributions, ROC surfaces, and pVUS regions (defined by $T{C}_1$ boundaries $p_1=0,p_2=0.2$, and $T{C}_3$ boundaries, $q_1=0.6,q_2=0.8$) for three PDAC biomarkers: LRG1, CA19.9, and MRPS24. The top three panels display the kernel-based probability density estimates for the Box-Cox transformed biomarker measurements. The shaded areas in these plots represent the range of cutoff points $c_1$ and $c_2$ that correspond to the restricted region. This restricted region is illustrated in the bottom three panels with the estimated ROC surfaces for each biomarker. Similar figures for the other four biomarkers studied are provided in Supplemental Appendix F. pVUS: partial volume under the receiver operating characteristic surface; PDAC: pancreatic ductal adenocarcinoma; ROC: receiver operating characteristic.