Computational modeling of human reasoning processes for interpretable visual knowledge: a case study with radiographers

Abstract

Visual reasoning is critical in many complex visual tasks in medicine such as radiology or pathology. It is challenging to explicitly explain reasoning processes due to the dynamic nature of real-time human cognition. A deeper understanding of such reasoning processes is necessary for improving diagnostic accuracy and computational tools. Most computational analysis methods for visual attention utilize black-box algorithms which lack explainability and are therefore limited in understanding the visual reasoning processes. In this paper, we propose a computational method to quantify and dissect visual reasoning. The method characterizes spatial and temporal features and identifies common and contrast visual reasoning patterns to extract significant gaze activities. The visual reasoning patterns are explainable and can be compared among different groups to discover strategy differences. Experiments with radiographers of varied levels of expertise on 10 levels of visual tasks were conducted. Our empirical observations show that the method can capture the temporal and spatial features of human visual attention and distinguish expertise level. The extracted patterns are further examined and interpreted to showcase key differences between expertise levels in the visual reasoning processes. By revealing task-related reasoning processes, this method demonstrates potential for explaining human visual understanding.

Figure 1

The system architecture of visual reasoning quantification and extraction.

Figure 2

Example eye movement sequences for Task 4. ROIs are marked with yellow lines. The connected blue lines and circles on each image show an entire eye movement sequence by one participant. Each circle is a fixation in the sequence, and its size represents the fixation duration. The red subsequences highlight the contrast pattern by experts which checks the R marks and the label. The sequences are plotted with Matplotlib.

Figure 3

Example eye movement sequences for Task 7. The red subsequences of experts show close examination of the potential diseased area, while the novices are distracted by the technical imperfections at the left edge (green subsequences).

Figure 4

Example eye movement sequences for Task 10. Unlike novices, the experts are not distracted by the non-anatomical structures and closely examine the fracture of the humeral head as shown by the red subsequences.