Higher-Order Conditioning With Simultaneous and Backward Conditioned Stimulus: Implications for Models of Pavlovian Conditioning

Abstract

In a new environment, humans and animals can detect and learn that cues predict meaningful outcomes, and use this information to adapt their responses. This process is termed Pavlovian conditioning. Pavlovian conditioning is also observed for stimuli that predict outcome-associated cues; a second type of conditioning is termed higher-order Pavlovian conditioning. In this review, we will focus on higher-order conditioning studies with simultaneous and backward conditioned stimuli. We will examine how the results from these experiments pose a challenge to models of Pavlovian conditioning like the Temporal Difference (TD) models, in which learning is mainly driven by reward prediction errors. Contrasting with this view, the results suggest that humans and animals can form complex representations of the (temporal) structure of the task, and use this information to guide behavior, which seems consistent with model-based reinforcement learning. Future investigations involving these procedures could result in important new insights on the mechanisms that underlie Pavlovian conditioning.

Keywords: backward conditioning; higher-order conditioning; reinforcement learning; reward prediction error; simultaneous conditioning.

Figure 1

Illustration of the second-order conditioning procedure. (A) Phase 1: First-order conditioning between a stimulus (CS1—sound) paired with an unconditioned stimulus (US—water). (B) Phase 2: Second-order conditioning between a second stimulus (CS2—light) paired with the previously paired stimulus CS1. (C) Classic results found in the second-order conditioning task with the conditioned response (CR) evoked both by CS1 and CS2. In sensory preconditioning, the procedure is similar except that phases 1 and 2 are inversed. (D) TD learning for the first-order conditioning phase with change in CS1’s predicted value V_CS1. Note that V_US is zero because of the absence of predicted value at the time of the US. Because R_US is positive, the pairing between CS1 and the US results in a positive δ (i.e., R_US − V_CS1 > 0), and the acquisition of predicted value from CS1 through the update of V_CS1 (V_CS1(new) = V_CS1(old) + α*δ). (E) TD learning for the second-order conditioning phase with change in CS2’s predicted value V_CS2. Note that R_CS1 is zero because of the absence of reward at CS1. Here, the positive V_CS1 learned during the first-order conditioning phase is sufficient to produce a positive δ (i.e., γ V_CS1 − V_CS2 > 0) and to increase the predicted value from CS2 (V_CS2). TD, Temporal Difference.

Figure 2

Illustration of simultaneous, backward, and second-order conditioning with simultaneous CS1. (A) Simultaneous conditioning with a stimulus (CS1—sound) presented simultaneously with an unconditioned stimulus (US—water). (B) Backward conditioning with CS1 presented after a US. (C) Second-order conditioning with simultaneous CS1: In phase 1, a stimulus CS1 is presented simultaneously with a US. In phase 2, a second stimulus CS2 is paired with CS1 through forward pairing. During the test, while CS1 will evoke low conditioned response (CR), CS2 will evoke substantial CR. According to the TD account, a low CR evoked by CS1 is expected because CS1 is not followed by the US (i.e., R_US = 0) in phase 1. In addition, a change in CS2 value (V_CS2) depends directly on CS1’s own value (V_CS1). Thus, a second-order pairing with a first-order stimulus CS1 that evokes a low CR level (and with presumably a low predicted value) should result in low responding to CS2. The evidence of substantial response to that stimulus challenges the TD account. The same holds for a model-based account of higher-order learning if the change in V_CS2 depends on CS1’s own predicted value V_CS1. Instead, it seems necessary for CS2’s predicted value to be based on US expectations to account for this finding. Note that the same pattern of results is observed for second-order conditioning with backward CS1, and for sensory preconditioning with simultaneous and backward CS1.