Biases introduced by choosing controls to match risk factors of cases in biomarker research

Abstract

Background: Selecting controls that match cases on risk factors for the outcome is a pervasive practice in biomarker research studies. Such matching, however, biases estimates of biomarker prediction performance. The magnitudes of these biases are unknown.

Methods: We examined the prediction performance of biomarkers and improvements in prediction gained by adding biomarkers to risk factor information. Data simulated from bivariate normal statistical models and data from a study to identify critically ill patients were used. We compared true performance with that estimated from case control studies that do or do not use matching. ROC curves were used to quantify performance. We propose a new statistical method to estimate prediction performance from matched studies for which data on the matching factors are available for subjects in the population.

Results: Performance estimated with standard analyses can be grossly biased by matching, especially when biomarkers are highly correlated with matching risk factors. In our studies, the performance of the biomarker alone was underestimated whereas the improvement in performance gained by adding the marker to risk factors was overestimated by 2-10-fold. We found examples for which the relative ranking of 2 biomarkers for prediction was inappropriately reversed by use of a matched design. The new approach to estimation corrected for bias in matched studies.

Conclusions: To properly gauge prediction performance in the population or the improvement gained by adding a biomarker to known risk factors, matched case control studies must be supplemented with risk factor information from the population and must be analyzed with nonstandard statistical methods.

Figure 1

ROC curves calculated with data from cases and unmatched controls and from a study where controls are selected to match cases in regards to clinical risk factor predictors (CP). Data for n=2000 cases and controls were simulated from bivariate binormal models corresponding to the scenario in Table 1 with ROC_X(0.2)= ROC_Y(0.2)=0.64 and correlation=0.5.

Figure 2

ROC curves calculated with data from cases and unmatched controls and from a study where controls are selected to match cases in regards to clinical risk factor predictors (CP). Data for n=100 cases and controls were simulated from bivariate binormal models corresponding to the scenario in Figure 1.

Figure 3

ROC curves for critical illness calculated using the clinical predictors alone (CP), the marker alone or using the marker and clinical predictors combined (CP+marker).

Figure 4

Comparison of markers in unmatched and matched studies. Left panels show performances of markers alone. Right panels show incremental value, i.e. performance of each marker combined with clinical predictors versus clinical predictors alone. The better marker is marker A in each scenario but it appears worse than marker B in a matched study.