Learning Classification Models of Cognitive Conditions from Subtle Behaviors in the Digital Clock Drawing Test

Abstract

The Clock Drawing Test - a simple pencil and paper test - has been used for more than 50 years as a screening tool to differentiate normal individuals from those with cognitive impairment, and has proven useful in helping to diagnose cognitive dysfunction associated with neurological disorders such as Alzheimer's disease, Parkinson's disease, and other dementias and conditions. We have been administering the test using a digitizing ballpoint pen that reports its position with considerable spatial and temporal precision, making available far more detailed data about the subject's performance. Using pen stroke data from these drawings categorized by our software, we designed and computed a large collection of features, then explored the tradeoffs in performance and interpretability in classifiers built using a number of different subsets of these features and a variety of different machine learning techniques. We used traditional machine learning methods to build prediction models that achieve high accuracy. We operationalized widely used manual scoring systems so that we could use them as benchmarks for our models. We worked with clinicians to define guidelines for model interpretability, and constructed sparse linear models and rule lists designed to be as easy to use as scoring systems currently used by clinicians, but more accurate. While our models will require additional testing for validation, they offer the possibility of substantial improvement in detecting cognitive impairment earlier than currently possible, a development with considerable potential impact in practice.

Keywords: Clock Drawing Test; Cognitive Impairment Diagnostics; Interpretable Machine Learning; Machine Learning Applications; Medical Scoring Systems.

Fig. 1

Summary of results for screening.

Fig. 2

Summary of results for diagnosis.

Fig. 3

Histogram of most frequent conditions in the data set.

Fig. 5

Example classified command clock from Figure 4b. An ellipse is fit to the clockface, with the major and minor axis shown; bounding boxes are drawn around each digit; arrows show the overall direction of the hands; the lines on digits 5, 10, and 12 show hooklets, with “x”s indicating the start of the next stroke after each hooklet. The system adds the colored overlays as a way of making stroke classification visually obvious.

Fig. 4

Example clocks, to scale, from our dataset for healthy controls, Alzheimer’s disease, and Parkinson’s disease, with command clock on the left and copy clock on the right

Fig. 6

A: the distance between starting and ending point of the clockface, as well as the angular difference; B: digit repetition; width and height of the bounding box; C: the difference in angle between a hand and its correct angle; D: hooklet presence, length, and direction.

Fig. 7

ROC curves for screening task (Table 1).

Fig. 8

ROC curves for the diagnosis task (Table 2).

Fig. 9

ROC curves for the experiments in Table 6.

Fig. 10

ROC curves for the experiments in Table 7.

Fig. 11

Scatter plot of Confidence vs. Support for rules for each condition vs. HC. Each dot on the plot represents an IF-THEN rule, where the condition is the THEN part of the rule. The right angle at the bottom left of each of these clusters shows the minimum confidence and support cutoffs used when mining the rules.

Fig. 12

Plot of AUC on testing folds vs. list length for simplest features, for both screening and diagnosis.