Tracking problem solving by multivariate pattern analysis and Hidden Markov Model algorithms

Abstract

Multivariate pattern analysis can be combined with Hidden Markov Model algorithms to track the second-by-second thinking as people solve complex problems. Two applications of this methodology are illustrated with a data set taken from children as they interacted with an intelligent tutoring system for algebra. The first "mind reading" application involves using fMRI activity to track what students are doing as they solve a sequence of algebra problems. The methodology achieves considerable accuracy at determining both what problem-solving step the students are taking and whether they are performing that step correctly. The second "model discovery" application involves using statistical model evaluation to determine how many substates are involved in performing a step of algebraic problem solving. This research indicates that different steps involve different numbers of substates and these substates are associated with different fluency in algebra problem solving.

Figure 1

Each panel illustrates one of the four steps in a problem-solving cycle with the tutor. The subpanels show the states of the tutor within a step. Each step starts with the last state of the previous step. The first panel starts with the initial equation x−10=17. Step 1: The student selects a transformation to perform on this equation by clicking on the two sides of the equation (resulting in the red highlighting) and choosing “Unwind” from the menu below. Step 2: The student expresses the result of the transformation by selecting the green box and entering 17 + 10. This results in the transformed equation x = 17+10. Step 3: The student specifies that 17+10 is to be evaluated by clicking on this expression (resulting in the red highlighting) and selecting “Evaluate” from the menu below. Step 4: The student specifies the result of the evaluation by entering 27. This creates the final answer x = 27, which is displayed here.

Figure 2

The behavioral model as a semi-Markov process. States correspond to steps (green correct, red incorrect) and rest period (R). While correct and incorrect states are numbered identically to indicate the step, each state is distinct within the HMM.

Figure 3

Figure 4. Differences between weights associated with error steps and correct steps. The 408 ROIs were created by evenly distributing 4×4×4 voxel cubes over the 34 slices of the 64×64 acquisition matrix. Between region spacing was 1 voxel in the x- and y- directions in the axial plane, and one slice in the z-direction. The final set of regions was acquired by applying a mask of the structural reference brain and excluding regions where less than 70% of the region's original 64 voxels survived.

Figure 4

Distribution of correct and error Step times for a student on Day 5 as a function of the difficulty other students experienced with that step. The points connected by dotted lines are the proportions of observations with different number of scans. The smooth lines are fitted gamma functions.

Figure 5

(a) Performance on segmentation. (b) Performance on diagnosis.

Figure 6

Step 1: Distribution of lengths of the two substates and their activation patterns. The activation values are z-scores for percent deviation from baseline.

Figure 7

Step 2: Distribution of lengths of the three substates and their activation patterns. The activation values are z-scores for percent deviation from baseline.

Figure 8

Step 3: Distribution of lengths of the one substates and its activation pattern. The activation values are z-scores for percent deviation from baseline.

Figure 9

Step 4: Distribution of lengths of two substates and their activation patterns. The activation values are z-scores for percent deviation from baseline.

Figure 10

Mean time in a substate as a function of day and whether an error was made on that step. Means are calculated separately for each student and averaged. The standard errors displayed are estimated from the student means.

Figure 11

Hierarchical clustering of substates: (a) Based on Euclidean distance between the 4 mean times associated with a substate (see Figure 10); (b) Based on Euclidean distance between the 408 voxel values m associated with a substate (see Figures 5–9).

Figure 12

Standard deviations of voxel values associated with the 8 substates.