Explainable automated evaluation of the clock drawing task for memory impairment screening

Abstract

Introduction: The clock drawing task (CDT) is frequently used to aid in detecting cognitive impairment, but current scoring techniques are time-consuming and miss relevant features, justifying the creation of an automated quantitative scoring approach.

Methods: We used computer vision methods to analyze the stored scanned images (N = 7,109), and an intelligent system was created to examine these files in a study of aging World Trade Center responders. Outcomes were CDT, Montreal Cognitive Assessment (MoCA) score, and incidence of mild cognitive impairment (MCI).

Results: The system accurately distinguished between previously scored CDTs in three CDT scoring categories: contour (accuracy = 92.2%), digits (accuracy = 89.1%), and clock hands (accuracy = 69.1%). The system reliably predicted MoCA score with CDT scores removed. Predictive analyses of the incidence of MCI at follow-up outperformed human-assigned CDT scores.

Discussion: We created an automated scoring method using scanned and stored CDTs that provided additional information that might not be considered in human scoring.

Keywords: Montreal Cognitive Assessment; World Trade Center responders; clock drawing task; semi‐automated neurocognitive testing.

FIGURE 1

Validation data needed to examine clock scoring reliability including correlation matrices, clock feature distributions, and example clock drawings separated by human‐assigned contour, digits, and hands scoring. Note: A–C show correlation matrices comparing contour features and human‐assigned scoring for (A) contours, (B) digits, and (C) hand scores; darker red identifies stronger associations that are more strongly positive, while deeper blue shows inverse associations. D–F show the cumulative distribution function for predicted (D) contour, (E) digits, and (F) hand scores; the two curves correspond to cohorts defined by the ground truth score assigned to the clock contour by human annotators (unsatisfactory = 0, and satisfactory = 1) and the area between curves shows reasonable separation between classes. G–I show annotated clock drawings representing predicted score percentiles in the lowest quartile, median, and top quartile for each (G) contour, (H) digits, and (I) hand human‐assigned scores; green labels the digits identified by the computer, purple identifies the hands, and yellow shows the circle that best fits the centroid.

FIGURE 2

Validation data needed to examine overall clock drawing task. Note: A shows the cumulative distribution function plot for overall clock drawing task scores. Ground truth scores defined by the total score received on the clock drawing task as determined by human annotators: total score of 0 (n = 20), total score of 1 (n = 141), total score of 2 (n = 532), and total score of 3 (n = 729). The curves demonstrate the expected separation, although all lower scores show a significant skew upward, probably because of the general data imbalance. B shows the box and whisker plot of the summed predicted clock scores compared with the adjusted total Montreal Cognitive Assessment (MoCA) score. Adjusted total MoCA score is provided by subtracting the human‐assigned clock score from the original total score. Regression lines of the mean predicted score (red) and the mean ground truth scores (blue) for each MoCA score group are shown superimposed over the plot.