100% classification accuracy considered harmful: the normalized information transfer factor explains the accuracy paradox

Abstract

The most widely spread measure of performance, accuracy, suffers from a paradox: predictive models with a given level of accuracy may have greater predictive power than models with higher accuracy. Despite optimizing classification error rate, high accuracy models may fail to capture crucial information transfer in the classification task. We present evidence of this behavior by means of a combinatorial analysis where every possible contingency matrix of 2, 3 and 4 classes classifiers are depicted on the entropy triangle, a more reliable information-theoretic tool for classification assessment. Motivated by this, we develop from first principles a measure of classification performance that takes into consideration the information learned by classifiers. We are then able to obtain the entropy-modulated accuracy (EMA), a pessimistic estimate of the expected accuracy with the influence of the input distribution factored out, and the normalized information transfer factor (NIT), a measure of how efficient is the transmission of information from the input to the output set of classes. The EMA is a more natural measure of classification performance than accuracy when the heuristic to maximize is the transfer of information through the classifier instead of classification error count. The NIT factor measures the effectiveness of the learning process in classifiers and also makes it harder for them to "cheat" using techniques like specialization, while also promoting the interpretability of results. Their use is demonstrated in a mind reading task competition that aims at decoding the identity of a video stimulus based on magnetoencephalography recordings. We show how the EMA and the NIT factor reject rankings based in accuracy, choosing more meaningful and interpretable classifiers.

Figure 1. Heatmap of the best classifiers of the MEG mind reading competition according to accuracy (left) and the EMA and the NIT factor (right) criteria.

Rows correspond to stimulus and columns to the decision or response. Darker hues correlate with higher joint probability . The heat map on the left reveals that the best classifier according to accuracy does not capture the fact that stimuli , and belong to a particular category whilst and belong to another. A B C

Figure 2. (Color online) Entropy decomposition for square matrices of (A) , (B) , and (C) (decimated), representing confusion matrices for a classification task at different accuracy levels as described by the right color bar.

The interspersing of the plots representing matrices with different accuracies but similar entropies is evident at all levels for and but only for lower levels of accuracy for . This entails that accuracy is not a good criterion to judge the flow of information from the input labels to the output labels of a classifier (see text).

Figure 3. (Color online) Entropy (above) and perplexity (below) decomposition chains for a joint distribution.

Left, perplexity reduction in the input (learning) chain; right, perplexity increase in the output chain, related to classifier specialization. The colors refer to those of Fig. 5.(B). The ordering of the boxes is a convention to reveal the prior and posterior natures of the perplexities of class distributions.

Figure 4. (Color online) Entropy triangle for the MEG mind Reading data ordered after accuracy (A) and a detail of the participants of higher accuracy (B).

The ranking following accuracy is at odds with the EMA and the NIT factor ranking based in mutual information (height, right scale of triangle). The detail in (B) shows that participant , closely followed by should have been ranked first after this criterion.

Figure 5. (Color online) Extended information diagrams of entropies related to a bivariate distribution: (A) conventional diagram, and (B) split diagram.

The bounding rectangle is the joint entropy of two uniform (thence independent) distributions and of the same cardinality as and . The expected mutual information appears twice in (A) and this makes the diagram split for each variable symmetrically in (B).

Figure 6. Schematic Entropy Triangle showing interpretable zones and extreme cases of classifiers.

The annotations on the center of each side are meant to hold for that whole side.