Imperfect gold standards for kidney injury biomarker evaluation

Abstract

Clinicians have used serum creatinine in diagnostic testing for acute kidney injury for decades, despite its imperfect sensitivity and specificity. Novel tubular injury biomarkers may revolutionize the diagnosis of acute kidney injury; however, even if a novel tubular injury biomarker is 100% sensitive and 100% specific, it may appear inaccurate when using serum creatinine as the gold standard. Acute kidney injury, as defined by serum creatinine, may not reflect tubular injury, and the absence of changes in serum creatinine does not assure the absence of tubular injury. In general, the apparent diagnostic performance of a biomarker depends not only on its ability to detect injury, but also on disease prevalence and the sensitivity and specificity of the imperfect gold standard. Assuming that, at a certain cutoff value, serum creatinine is 80% sensitive and 90% specific and disease prevalence is 10%, a new perfect biomarker with a true 100% sensitivity may seem to have only 47% sensitivity compared with serum creatinine as the gold standard. Minimizing misclassification by using more strict criteria to diagnose acute kidney injury will reduce the error when evaluating the performance of a biomarker under investigation. Apparent diagnostic errors using a new biomarker may be a reflection of errors in the imperfect gold standard itself, rather than poor performance of the biomarker. The results of this study suggest that small changes in serum creatinine alone should not be used to define acute kidney injury in biomarker or interventional studies.

Figure 1.

Apparent diagnostic performance characteristics of a perfect biomarker compared with an imperfect gold standard. (A) Actual disease prevalence = 20%. Red squares represent true disease positives. Blue squares represent true disease negatives. (B) Imperfect gold standard sensitivity = 80%. Of the 20 true disease positives, 4 are classified as disease negative by the imperfect gold standard (darker blue). (C) Imperfect gold standard specificity = 80%. Of the 80 true disease negatives, 16 are classified as disease positive by the imperfect gold standard (darker red). (D) The imperfect gold standard has classified 32 individuals as disease positive (red) and 68 individuals as disease negative (blue). A perfect biomarker will correctly identify only 16 of the original true positives (lighter red) and fail to identify the 16 false positives (darker red), leading to an apparent sensitivity of 16/32 = 50%. A perfect biomarker will identify as disease negative only 64 of the original true negatives (lighter blue) and fail to identify the 4 false negatives (darker blue), leading to an apparent specificity of 64/68 = 94%.

Figure 2.

Apparent diagnostic performance characteristics of a perfect biomarker when an imperfect gold standard has 80% sensitivity and 90% specificity. (A) The apparent sensitivity and specificity. (B) The apparent AUC-ROC curve (solid line, under the assumption of conditional independence) with lower and upper bounds (dotted lines). Results are plotted for a range of true disease prevalence estimates.

Figure 3.

Apparent diagnostic performance characteristics of a perfect biomarker when an imperfect gold standard has 25% sensitivity and 100% specificity. (A) The apparent sensitivity and specificity. (B) The apparent AUC-ROC curve (solid line, under the assumption of conditional independence) with lower and upper bounds (dotted lines). Results are plotted for a range of true disease prevalence estimates.

Figure 4.

AUC-ROC curve of a perfect biomarker when an imperfect gold standard has variable sensitivity or specificity, as shown. Results are plotted for a true disease prevalence of 10%.