Neural imaging to track mental states while using an intelligent tutoring system

Abstract

Hemodynamic measures of brain activity can be used to interpret a student's mental state when they are interacting with an intelligent tutoring system. Functional magnetic resonance imaging (fMRI) data were collected while students worked with a tutoring system that taught an algebra isomorph. A cognitive model predicted the distribution of solution times from measures of problem complexity. Separately, a linear discriminant analysis used fMRI data to predict whether or not students were engaged in problem solving. A hidden Markov algorithm merged these two sources of information to predict the mental states of students during problem-solving episodes. The algorithm was trained on data from 1 day of interaction and tested with data from a later day. In terms of predicting what state a student was in during a 2-s period, the algorithm achieved 87% accuracy on the training data and 83% accuracy on the test data. The results illustrate the importance of integrating the bottom-up information from imaging data with the top-down information from a cognitive model.

Fig. 1.

Sequences of tutor interaction during a problem isomorph. (A) The student starts out in a state with a data-flow equivalent of the equation x – 10 = 17. The student uses the mouse to select this equation and chooses the operation “Invert” from the menu. (B) A keypad comes up into which the student enters the result 17 + 10. (C) The transformation is complete. (D) The previous state (data-flow equivalent of x = 17 + 10) is repeated and the student selects 17 + 10 and chooses the operation “Evaluate”. (E) A keypad comes up into which the student will type “27.” (F) The evaluation is complete.

Fig. 2.

Systematic relationship that exists between mouse clicks required to accomplish an operation and the time that the operation took. (A and C) The relationship between number of clicks and number of scans. (B and D) Distributions of number of scans for different numbers of clicks and log-normal distributions fitted to these.

Fig. 3.

Accuracy of classification and distribution of values. (A) Accuracy of classification as a function of the offset between the scan whose activity is being used and the scan whose state is being predicted. (B) Distribution of values for Day 1 and Day 5 On and Off scans using an offset of 2. All 408 regions are used.

Fig. 4.

The 48 most predictive regions. These regions result in a d-prime of 2.48 on the training data and 2.23 on the test data when their activity is used to predict the state of two scans earlier. The regions are color-coded based on their weights.

Fig. 5.

An example of an experimental block and its interpretations. The sequence of equations is shown in column A. Columns B, C, and D compare attempts at predicting the states with both fMRI and model, just fMRI, or just model. On scans (when an equation is on the screen) are to the Left and Off times (when no equation is on the screen) are to the Right.

Fig. 6.

Performance, measured as the distance between the actual state and the predicted state, using both cognitive model and fMRI, just fMRI, or just a cognitive model on (A) Day 1 and (B) Day 5.