Modelling multiple thresholds in meta-analysis of diagnostic test accuracy studies

Abstract

Background: In meta-analyses of diagnostic test accuracy, routinely only one pair of sensitivity and specificity per study is used. However, for tests based on a biomarker or a questionnaire often more than one threshold and the corresponding values of true positives, true negatives, false positives and false negatives are known.

Methods: We present a new meta-analysis approach using this additional information. It is based on the idea of estimating the distribution functions of the underlying biomarker or questionnaire within the non-diseased and diseased individuals. Assuming a normal or logistic distribution, we estimate the distribution parameters in both groups applying a linear mixed effects model to the transformed data. The model accounts for across-study heterogeneity and dependence of sensitivity and specificity. In addition, a simulation study is presented.

Results: We obtain a summary receiver operating characteristic (SROC) curve as well as the pooled sensitivity and specificity at every specific threshold. Furthermore, the determination of an optimal threshold across studies is possible through maximization of the Youden index. We demonstrate our approach using two meta-analyses of B type natriuretic peptide in heart failure and procalcitonin as a marker for sepsis.

Conclusions: Our approach uses all the available information and results in an estimation not only of the performance of the biomarker but also of the threshold at which the optimal performance can be expected.

Keywords: Biomarker; Diagnostic accuracy study; Meta-analysis; ROC curve; Threshold.

Fig. 1

Bias of sensitivity and specificity at threshold 0 (top panel) and at the true optimal threshold (bottom panel). The Poisson distribution was chosen with λ=1.3 for the number of thresholds. The four plots at the bottom show the case of same standard deviations (SD), the top four plots the case of different standard deviations. The heterogeneity of the studies increases from left to right

Fig. 2

Coverage of sensitivity and specificity at the true optimal threshold (top panel). Bias of the optimal threshold (bottom panel). The Poisson distribution was chosen with λ=1.3 for the number of thresholds. The four plots at the bottom show the case of same standard deviations (SD), the top four plots the case of different standard deviations. The heterogeneity of the studies increases from left to right

Fig. 3

Percentage of error messages (left panel) and warnings (right panel) in simulation scenarios with a varying number of thresholds. Red circles indicate cases with negative regression slope

Fig. 4

Percentage of error messages (left panel) and warnings (right panel) in simulation scenarios with the number of thresholds fixed to five. Red circles indicate cases with negative regression slope

Fig. 5

B type natriuretic peptide data. a Estimated distribution functions for the non-diseased (open circles, dashed line) and the diseased individuals (filled circles, solid line). The grey lines mark the confidence regions and different studies are marked in different colours. The optimal threshold, derived from a maximization of the Youden index, is depicted as a solid vertical line. b Estimated densities and their point of intersection. Non-diseased (dashed line), diseased individuals (solid line). c Study-specific ROC curves. d Estimated SROC curve with the optimal thresholds for different weightings of sensitivity and specificity marked as crosses in black, red and green. Different studies are marked in different colours

Fig. 6

Procalcitonin data. a Estimated distribution functions for the non-diseased (open circles, dashed line) and the diseased individuals (filled circles, solid line). The grey lines mark the confidence regions and different studies are marked in different colours. The optimal threshold, derived from a maximization of the Youden index, is depicted as a solid vertical line. b Estimated densities and their point of intersection. Non-diseased (dashed line), diseased individuals (solid line). c Study-specific ROC curves. d Estimated SROC curve with the optimal threshold for equal weighting of sensitivity and specificity. Different studies are marked in different colours