Tracking children's mental states while solving algebra equations

Abstract

Behavioral and function magnetic resonance imagery (fMRI) data were combined to infer the mental states of students as they interacted with an intelligent tutoring system. Sixteen children interacted with a computer tutor for solving linear equations over a six-day period (days 0-5), with days 1 and 5 occurring in an fMRI scanner. Hidden Markov model algorithms combined a model of student behavior with multi-voxel imaging pattern data to predict the mental states of students. We separately assessed the algorithms' ability to predict which step in a problem-solving sequence was performed and whether the step was performed correctly. For day 1, the data patterns of other students were used to predict the mental states of a target student. These predictions were improved on day 5 by adding information about the target student's behavioral and imaging data from day 1. Successful tracking of mental states depended on using the combination of a behavioral model and multi-voxel pattern analysis, illustrating the effectiveness of an integrated approach to tracking the cognition of individuals in real time as they perform complex tasks.

Figure 1

Each panel illustrates one of the four steps in a problem‐solving cycle with the tutor. The subpanels show the states of the tutor within a step. Each step starts with the last state of the previous step. The subpanels also indicate the minimal number of mouse clicks required to achieve that state for this specific problem. The first panel starts with the initial equation x − 10 = 17. Step 1: The student selects a transformation to perform on this equation by clicking on the two sides of the equation (resulting in red highlighting) and choosing “Unwind” from the menu below. “Unwind” refers to undoing the operations surrounding the unknown; in this case undoing the “−10” by adding 10. Step 2: The student expresses the result of the transformation by selecting a green box and entering 17 + 10. This results in the transformed equation x = 17 + 10. Step 3: The student specifies that 17 + 10 is to be evaluated by clicking on this expression (resulting in the highlighting) and selecting “Evaluate” from the menu below. Step 4: The student specifies the result of the evaluation by selecting a green box and entering 27. This creates the final answer x = 27. [Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.]

Figure 2

Material presented over days and structure of scanning blocks on Days 1 and 5. The algebraic expressions are examples of what appeared in various sections on scanning days.

Figure 3

(a,b): Distribution of step times for a student on Day 1 as function of the difficulty other students experienced with that step. (c,d): Distribution of step times for a student on Day 5 as function of the difficulty other students experienced with that step and how slow that student was on Day 1. The points connected by dotted lines are the actual proportions of observations with different number of scans. The smooth lines are fitted gamma functions. [Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.]

Figure 4

Probability that a student will make an error on a step as a function of the predictor error estimated from other data. Similar predictor error rates are aggregated to give different points on the figure. [Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.]

Figure 5

Regions showing significant effects of error (a), transformation versus evaluation (b), and selection versus execution (c). Refer to Figure 1 for illustration of transformation, evaluation, selection, and execution. Values indicated by color are t values.

Figure 6

Ability of the linear discriminant function to distinguish among categories. The x‐axis gives the various categories and proportion of scans from that category. There were not enough observations of errors on step 3 to allow meaningful statistics (less than 0.5% on Day 1 and less than 0.2% on Day 5). The bars for each category show the proportion of scans in each category assigned to each of the nine possible categories.

Figure 7

The 55 highly discriminative regions, labeled with the feature for which they were most active. Highlighted are the regions that are particularly striking and which suggest possible interpretations. They are labeled with their anatomical region and with the name of the ACT‐R module that they have been found to correlate with in past models of algebraic information processing [e.g., Anderson, 2005; Anderson et al., 2008].

Figure 8

An example of an experimental block and various attempts to assign scans to stages of problem solving. The x‐axis gives the scan number and the y‐axis displays the progress of the student through seven problems (on Day 1, involving collection of constants—actual problems given in figure) starting in a rest state (0) and stepping through four states. The green dotted line indicates correctly performed steps and the red dotted line indicates incorrectly performed steps. The blue line displays the student's true trajectory and the orange line displays the assigned trajectory. (a) Scans are assigned to the most probable state based on the conditional probabilities from the linear discriminant analysis. (b) The behavioral model and imaging analysis are combined by the Forward HMM algorithm to make the best real‐time assignments. (c) The behavioral and imaging analyses are combined by the Viterbi HMM algorithm to make the best assignments after the block has ended. (d) Only fMRI data are used with the Viterbi Algorithm. (e) Only the behavioral model is used with the Viterbi Algorithm.

Figure 9

Representation of the behavioral model as a semi‐Markov process. States correspond to steps (green for correct, red for incorrect) and rest period (R).

Figure 10

Performance of the Forward Algorithm with respect to the segmentation goal. The x‐axis gives the distance between the actual step and the predicted step: (a) Day 1 and (b) Day 5.

Figure 11

The performance on error classification of steps using the segmentation inferred by the Viterbi Algorithm from the various information sources. Blue is Day 1 performance and red is Day 5 performance. The ROC curves assume correct segmentation and use both the linear discriminant analysis of the imaging data and the behavioral model of the behavioral data.