Measuring the accuracy of prognostic judgments in oncology

Abstract

The purpose of this study was to describe the accuracy of prognostic judgments in the day-to-day practice of cancer medicine. Oncologists' initial estimates of the probability of cure of 98 cancer patients, and of the duration of survival of 39 incurable cancer patients, were compared with the observed outcome at five years. Estimates of probability of cure were well calibrated and also relatively accurate (Mean Brier Accuracy Score [AS] = 0.11 +/- 0.02 [SE]). Oncologists' ability to discriminate between curable and incurable patients compared favorably with the discrimination of prognostic judgments in acute care medicine (area under the receiver operating characteristic curve, AROC = 0.91 +/- 0.09). Estimates of the duration of survival of incurable patients were well calibrated, but individual predictions were imprecise. Discrimination between patients who would survive for three months, and those who would not, was only fair (AROC = 0.75 +/- 0.04), and discrimination between patients who would survive for one year and those who would not was very poor (AROC = 0.57 +/- 0.01). It was concluded that oncologists' estimates of probability of cure are surprisingly accurate, but estimates of the duration of survival of incurable patients are much less accurate, and subject to a "horizon effect" similar to that recognized in weather forecasting.