The Matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation

Abstract

Evaluating binary classifications is a pivotal task in statistics and machine learning, because it can influence decisions in multiple areas, including for example prognosis or therapies of patients in critical conditions. The scientific community has not agreed on a general-purpose statistical indicator for evaluating two-class confusion matrices (having true positives, true negatives, false positives, and false negatives) yet, even if advantages of the Matthews correlation coefficient (MCC) over accuracy and F₁ score have already been shown.In this manuscript, we reaffirm that MCC is a robust metric that summarizes the classifier performance in a single value, if positive and negative cases are of equal importance. We compare MCC to other metrics which value positive and negative cases equally: balanced accuracy (BA), bookmaker informedness (BM), and markedness (MK). We explain the mathematical relationships between MCC and these indicators, then show some use cases and a bioinformatics scenario where these metrics disagree and where MCC generates a more informative response.Additionally, we describe three exceptions where BM can be more appropriate: analyzing classifications where dataset prevalence is unrepresentative, comparing classifiers on different datasets, and assessing the random guessing level of a classifier. Except in these cases, we believe that MCC is the most informative among the single metrics discussed, and suggest it as standard measure for scientists of all fields. A Matthews correlation coefficient close to +1, in fact, means having high values for all the other confusion matrix metrics. The same cannot be said for balanced accuracy, markedness, bookmaker informedness, accuracy and F₁ score.

Keywords: Balanced accuracy; Binary classification; Bookmaker informedness; Confusion matrix; Machine learning; Markedness; Matthews correlation coefficient.

Fig. 1

Relationships between MCC and BM, BA, and MK. Plots indicating the values of MCC in relationship with BM (left), BA (centre), and MK (right) calculated for approximately 8 million confusion matrices with 40 thousand samples each

Fig. 2

Pearson correlation between MCC, BM and MK as a function of number of samples N

Fig. 3

Indicative example with high true positive rate (TPR) and high true negative rate (TNR). We show the trend of the four basic rates if TPR = 0.9 and TNR = 0.8, and illustrate how positive predictive value (PPV) and negative predictive value (NPV) depend on prevalence (ϕ). Bookmaker informedness (BM) equals 0.7 in this example. At least one of PPV and NPV is high, even if ϕ varies. Only if ϕ is close to 0.6, both of them are high

Fig. 4

BM, MCC and MK for classifiers with known randomness. We simulated classifiers with known amounts of randomness. To that purpose, we generated random lists of reference classes with a given prevalence. A fraction of those classes were copied (called lookup fraction) and used as predicted labels, the remaining ones were generated randomly, matching a given bias. Matching the reference classes with the prediction labels we determined bookmaker informedness, Matthews correlation coefficient and markedness (left, center and right column respectively). The rows differ by the amount of randomness/lookup fraction