On criteria for evaluating models of absolute risk

Abstract

Absolute risk is the probability that an individual who is free of a given disease at an initial age, a, will develop that disease in the subsequent interval (a, t]. Absolute risk is reduced by mortality from competing risks. Models of absolute risk that depend on covariates have been used to design intervention studies, to counsel patients regarding their risks of disease and to inform clinical decisions, such as whether or not to take tamoxifen to prevent breast cancer. Several general criteria have been used to evaluate models of absolute risk, including how well the model predicts the observed numbers of events in subsets of the population ("calibration"), and "discriminatory power," measured by the concordance statistic. In this paper we review some general criteria and develop specific loss function-based criteria for two applications, namely whether or not to screen a population to select subjects for further evaluation or treatment and whether or not to use a preventive intervention that has both beneficial and adverse effects. We find that high discriminatory power is much more crucial in the screening application than in the preventive intervention application. These examples indicate that the usefulness of a general criterion such as concordance depends on the application, and that using specific loss functions can lead to more appropriate assessments.