Calibrating random forests for probability estimation

Abstract

Probabilities can be consistently estimated using random forests. It is, however, unclear how random forests should be updated to make predictions for other centers or at different time points. In this work, we present two approaches for updating random forests for probability estimation. The first method has been proposed by Elkan and may be used for updating any machine learning approach yielding consistent probabilities, so-called probability machines. The second approach is a new strategy specifically developed for random forests. Using the terminal nodes, which represent conditional probabilities, the random forest is first translated to logistic regression models. These are, in turn, used for re-calibration. The two updating strategies were compared in a simulation study and are illustrated with data from the German Stroke Study Collaboration. In most simulation scenarios, both methods led to similar improvements. In the simulation scenario in which the stricter assumptions of Elkan's method were not met, the logistic regression-based re-calibration approach for random forests outperformed Elkan's method. It also performed better on the stroke data than Elkan's method. The strength of Elkan's method is its general applicability to any probability machine. However, if the strict assumptions underlying this approach are not met, the logistic regression-based approach is preferable for updating random forests for probability estimation. © 2016 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd.

Keywords: calibration; logistic regression; probability estimation; probability machine; random forests; updating.

Figure 1

Example of a probability estimation tree for a dichotomous outcome y given covariates x ₁ and x ₂. Split points are c ₁ and c ₂, and the terminal nodes are labeled t ₁, t ₂, and t ₃.

Figure 2

Steps in logistic regression‐based updating approach for random forests.

Figure 3

True versus predicted probabilities for simulation scenario 1. RF: random forest built on model‐building data; RF + LogReg: RF translated to logistic regression models; RF + LogReg + Cal: RF + LogReg updated using re‐calibration; RF + CalElkan: probabilities from RF updated using Elkan's method; RF onCalData: random forest built on cal‐training data; LogReg + Cal: logistic regression fitted using model‐building data and updated using re‐calibration.

Figure 4

Mean‐squared errors between true and predicted probabilities in the simulation study. RF: random forest built on model‐building data; RF + LogReg: RF translated to logistic regression models; RF + LogReg + Cal: RF + LogReg updated using re‐calibration; RF + CalElkan: probabilities from RF updated using Elkan's method; RF onCalData: random forest built on cal‐training data; LogReg + Cal: logistic regression fitted using model‐building data and updated using re‐calibration.

Figure 5

Mean‐squared errors between true and predicted probabilities in simulation scenario 5, where the calibration data for training were substantially smaller than in the other simulation scenarios. RF: random forest built on model‐building data; RF + LogReg: RF translated to logistic regression models; RF + LogReg + Cal: RF + LogReg updated using re‐calibration; RF + CalElkan: probabilities from RF updated using Elkan's method; RF onCalData: random forest built on cal‐training data; LogReg + Cal: logistic regression fitted using model‐building data and updated using re‐calibration.

Figure 6

True versus predicted probabilities for simulation scenario 6. RF: random forest built on model‐building data; RF + LogReg: RF translated to logistic regression models; RF + LogReg + Cal: RF + LogReg updated using re‐calibration; RF + CalElkan: probabilities from RF updated using Elkan's method; RF onCalData: random forest built on cal‐training data; LogReg + Cal: logistic regression fitted using model‐building data and updated using re‐calibration.

Figure 7

Calibration curves for external stroke validation data. RF: random forest built on training data; RF + CalElkan: probabilities from RF updated using Elkan's method; RF + LogReg + Cal: RF translated to logistic regression models and updated using re‐calibration.