Bias in estimating accuracy of a binary screening test with differential disease verification

Abstract

Sensitivity, specificity, positive and negative predictive value are typically used to quantify the accuracy of a binary screening test. In some studies, it may not be ethical or feasible to obtain definitive disease ascertainment for all subjects using a gold standard test. When a gold standard test cannot be used, an imperfect reference test that is less than 100 per cent sensitive and specific may be used instead. In breast cancer screening, for example, follow-up for cancer diagnosis is used as an imperfect reference test for women where it is not possible to obtain gold standard results. This incomplete ascertainment of true disease, or differential disease verification, can result in biased estimates of accuracy. In this paper, we derive the apparent accuracy values for studies subject to differential verification. We determine how the bias is affected by the accuracy of the imperfect reference test, the percent who receive the imperfect reference standard test not receiving the gold standard, the prevalence of the disease, and the correlation between the results for the screening test and the imperfect reference test. It is shown that designs with differential disease verification can yield biased estimates of accuracy. Estimates of sensitivity in cancer screening trials may be substantially biased. However, careful design decisions, including selection of the imperfect reference test, can help to minimize bias. A hypothetical breast cancer screening study is used to illustrate the problem.

Figure 1

Diagram illustrating IDV and CDV designs.

Figure 2

Percent bias in sensitivity (solid curve) and specificity (dashed curve) for CDV design where the imperfect reference test is 100% specific but less than 100% sensitive. The percentage of results misclassified is varied. Prevalence is 10% and Sp_T = 0.7.

Figure 3

Percent bias in sensitivity resulting from CDV designs. Left panels have true prevalence of 10% while the right panels have prevalence of 30%. Top row: Se_T=0.5, Sp_T=0.8, Se_R=0.7, Sp_R=0.9; Middle row: Se_T=0.7, Sp_T=0.9, Se_R=0.7, Sp_R=0.9; Bottom row: Se_T=0.9, Sp_T=0.9, Se_R=0.7, Sp_R=0.7. Minimum TNNF (solid line), median TNNF (dashed line), and maximum TNNF (dotted line).

Figure 4

Percent bias in sensitivity resulting from IDV designs with τ = 0. Left panels have true prevalence of 10% while the right panels have prevalence of 30%. Top row: Se_T=0.5, Sp_T=0.8, Se_R=0.7, Sp_R=0.9, TPPF=0.3; Middle row: Se_T=0.7, Sp_T=0.9, Se_R=0.7, Sp_R=0.9, TPPF=0.5; Bottom row: Se_T=0.9, Sp_T=0.9, Se_R=0.7, Sp_R=0.7, TPPF=0.65. Minimum TNNF (solid line), median TNNF (dashed line), and maximum TNNF (dotted line).

Figure 5

Percent bias in PPV resulting from IDV designs with τ = 0. Left panels have true prevalence of 10% while the right panels have prevalence of 30%. Top row: Se_T=0.5, Sp_T=0.8, Se_R=0.7, Sp_R=0.9, TPPF=0.3; Middle row: Se_T=0.7, Sp_T=0.9, Se_R=0.7, Sp_R=0.9, TPPF=0.5; Bottom row: Se_T=0.9, Sp_T=0.9, Se_R=0.7, Sp_R=0.7, TPPF=0.65. Minimum TNNF (solid line), median TNNF (dashed line), and maximum TNNF (dotted line).