A Framework for Considering Comprehensibility in Modeling

Abstract

Comprehensibility in modeling is the ability of stakeholders to understand relevant aspects of the modeling process. In this article, we provide a framework to help guide exploration of the space of comprehensibility challenges. We consider facets organized around key questions: Who is comprehending? Why are they trying to comprehend? Where in the process are they trying to comprehend? How can we help them comprehend? How do we measure their comprehension? With each facet we consider the broad range of options. We discuss why taking a broad view of comprehensibility in modeling is useful in identifying challenges and opportunities for solutions.

Keywords: data analysis; human-computer interaction; machine learning; statistical modeling; visual analytics; visualization.

FIG. 1.

A summary of the framework proposed in this article. The specific lists for each question are initial organizations to show the broad range of aspects to consider.

FIG. 2.

Visualization of a validation experiment for a DNA-binding surface classifier that allows exploration of classification results. The corpus overview (left) is configured to display each molecule in the test set as a quilted glyph and orders these glyphs by classifier performance to show how performance varies over the molecules. Those proteins that appear more green have more true positive classifications, whereas those molecular that appear more red or blue have more misclassifications (false negatives and false positives, respectively). Selected molecules (left, yellow box) are visualized as heatmaps in a subset view (middle) and ordered by molecule size to help localize the positions of errors relative to correct answers. The detailed view (right) shows a selected molecule to confirm that most errors (blue, red) are close to the correctly found binding site (green).

FIG. 3.

Visualization of example Explainers, classifiers constructed with tradeoffs that emphasize comprehensibility concerns. In this example, Shakepeare's 36 plays are measured with a set of 115 “Docuscope” features. Classifiers are constructed to identify the 12 comedies (green). Each column represents a linear SVM classifier, with the plays sorted according to their score. The leftmost classifier uses only two features with unit coefficients. It makes several mistakes (e.g., misclassifying the tragedies Othello and Romeo and Juliet as comedies), but the simplicity of the classifier makes it useful for building theory about how Shakespeare used the linguistic constructs in the different genres. In contrast, other classifiers may use more features and more complex weights to achieve better accuracy (and larger SVM margins), at the expense of how easy the functions are to comprehend. SVM, support vector machine.