Evaluating prediction model performance

Abstract

This article highlights important performance metrics to consider when evaluating models developed for supervised classification or regression tasks using clinical data. When evaluating model performance, we detail the basics of confusion matrices, receiver operating characteristic curves, F1 scores, precision-recall curves, mean squared error, and other considerations. In this era, defined by the rapid proliferation of advanced prediction models, familiarity with various performance metrics beyond the area under the receiver operating characteristic curves and the nuances of evaluating model value upon implementation is essential to ensure effective resource allocation and optimal patient care delivery.

Figure 1.. Confusion Matrix.

This matrix allows for calculation of key model metrics such as sensitivity/recall, specificity, precision, and accuracy.

Figure 2:. Calibration Plot.

These plots provide a visual example of predicted probablity relative to event rate in a collection of samples. Samples are divided into 10 bins based on their predicted probablity([0–10%], [10–20%], …). For each bin, the percentage of positive events is plotted on the y-axis relative to the center of each bin on the x-axis. The diagnol dashed line represents a perfectly calibrated model for reference.