Beware of R(2): Simple, Unambiguous Assessment of the Prediction Accuracy of QSAR and QSPR Models

Abstract

The statistical metrics used to characterize the external predictivity of a model, i.e., how well it predicts the properties of an independent test set, have proliferated over the past decade. This paper clarifies some apparent confusion over the use of the coefficient of determination, R(2), as a measure of model fit and predictive power in QSAR and QSPR modeling. R(2) (or r(2)) has been used in various contexts in the literature in conjunction with training and test data for both ordinary linear regression and regression through the origin as well as with linear and nonlinear regression models. We analyze the widely adopted model fit criteria suggested by Golbraikh and Tropsha ( J. Mol. Graphics Modell. 2002 , 20 , 269 - 276 ) in a strict statistical manner. Shortcomings in these criteria are identified, and a clearer and simpler alternative method to characterize model predictivity is provided. The intent is not to repeat the well-documented arguments for model validation using test data but rather to guide the application of R(2) as a model fit statistic. Examples are used to illustrate both correct and incorrect uses of R(2). Reporting the root-mean-square error or equivalent measures of dispersion, which are typically of more practical importance than R(2), is also encouraged, and important challenges in addressing the needs of different categories of users such as computational chemists, experimental scientists, and regulatory decision support specialists are outlined.

Figure 1

For the black data alone, R² = 0.49, while for the combined data the regression line is the same but R² = 0.79. The RMSE is the same in each case, as the red and black residuals are identical, but increasing the range of activity values in the data increases R².

Figure 2

Illustrative plots of observed and predicted data. The dotted line shows the relationship y = ŷ data points for good models would lie close to this line. Solid regression lines and dashed lines for regression through the origin are also added in order to illustrate the test data conditions of Golbraikh and Tropsha, though plotting such lines on the graphs is not advocated here.

Figure 3

The test set predictions (154 compounds in test set) from a model predicting the melting point of 771 ionic liquids (ionic liquid data from Varnek et al.) A regression line has been drawn through the data (solid line). The dotted line is for x=y and the regression values relate to fitting points to each line. The two graphs represent the plotting of the same test set data but with reversal of the assignment of the observed melting points to the axes. Note that the x=y regression gives markedly different results depending on which data are assigned to the y-axis.