Prediction error associated with the perceptual segmentation of naturalistic events

Abstract

Predicting the near future is important for survival and plays a central role in theories of perception, language processing, and learning. Prediction failures may be particularly important for initiating the updating of perceptual and memory systems and, thus, for the subjective experience of events. Here, we asked observers to make predictions about what would happen 5 sec later in a movie of an everyday activity. Those points where prediction was more difficult corresponded with subjective boundaries in the stream of experience. At points of unpredictability, midbrain and striatal regions associated with the phasic release of the neurotransmitter dopamine transiently increased in activity. This activity could provide a global updating signal, cuing other brain systems that a significant new event has begun.

Figure 1.

Illustration of the selection of within-event and across-event trials. The black curve plots the probability density of segmentation for the LEGO movie. The red arrows indicate the across-event condition test points, and the blue arrows indicate the within-event condition test points.

Figure 2.

Predicting the near future is more difficult at event boundaries. a. Participants viewed movies of everyday events (247s - 432s duration). Eight times during each movie, the movie was paused and the viewer was asked to predict what would be on the screen in 5 s. Experiments 1 and 3 used a 2-alternative forced choice procedure in which participants chose between the correct picture and a similar-appearing picture taken from a different movie. (In this example the picture on the left is correct.) Experiment 2 used a yes-no procedure in which one of the two possible pictures was presented and participants were required to assess whether it was correct. In Experiment 3 participants made an assessment of their prediction confidence using a 6-point Likert-type scale before the two test pictures were presented. b. By all measures, prediction performance was worse for across-event trials than within-event trials. Statistical tests were conducted using both participants and items as the random effect. In Experiments. 1, 3, and 4, forced-choice accuracy was lower (top graph) and responses were slower (second graph) when predicting across events. [By participants: smallest t(23) = 6.68, p < .001; by items: smallest t(38) = 2.84, p = .007. For response time, by participants: smallest t(23) = 3.28, p = .003; by items: smallest t(38) = 2.42, p = .007.)]Experiment 2 was analyzed using signal detection analysis (Macmillan & Creelman, 1991) which showed that discrimination between correct and incorrect pictures was lower across than within events (third graph; by participants, t(23) = 4.82, p < .001; by items, t(38) = 3.56, p = .001). Moreover, participants’ decision criteria were more conservative across than within events, indicating that, overall, pictures taken from new events were judged less likely to be correct, though this effect was significant only by participants (fourth graph; by participants, t₂₃ = 2.71, p = .013; by items, t(38) = 0.98, p = .335, NS). In Experiment 3, confidence judgments made before presentation of the test pictures indicated that participants were less confident in their ability to predict as they approached a new event (bottom graph; by participants, t(23) = 6.81, p < .001; by items, t(38) = 2.42, p = .021. (Error bars depict standard errors.)

Figure. 3.

The MDS and its striatal targets are activated when attempting to predict features of a new event. a. For each participant, the SN and VTAs were identified by manual tracing using Caret (Van Essen et al., 2001), and the caudate nucleus and putamen were identified using FreeSurfer (Jovicich et al., 2009) (see Supplementary Methods). b. Magnitude of evoked fMRI response when attempting to predict. Significant responses during the prediction task were observed in the left SN, caudate and putamen. For the right SN, activity was significantly greater for prediction across than within events (red bracket); for the right caudate this trend approached statistical significance (gray bracket). (Error bars depict standard errors.)

Figure. 4.

Disconfirming information from the beginning of a new event was accompanied by increased fMRI response in the MDS and its striatal targets. Each panel plots the difference in the fMRI response to restarting the movie in an across-event trial compared to a within-event trial. The new event’s beginning was 2.5 s after the movie onset. Between 8 and 10 s after the movie onset, a maximal difference response is observed. (Gray regions depict standard errors.)

Figure 5.

A whole-brain analysis found regions of parietotemporal cortex that were activated when attempting to predict features of a new event. Regions of significant activation are shown projected on left and right lateral views of the cortical hemispheres using Caret (Van Essen et al., 2001).

Figure 6.

Disconfirming information from the beginning of a new event was accompanied by increased fMRI response throughout the cortex. A. Areas with a significant difference in the timecourse of response after the onset of the movie in the within- and across-event conditions, projected on the medial and lateral cortical hemispheres using Caret (Van Essen et al., 2001). The two regions that showed the strongest effect are highlighted and plotted in B and C. B and C plot the difference in the fMRI response to restarting the movie in an across-event trial compared to a within-event trial in each region. The new event’s beginning was 2.5 s after the movie onset. Between 8 and 10 s after the movie onset, a maximal difference response is observed, similar to that observed in the midbrain and striatum (see Figure 4). (Gray regions depict standard errors.)