Choice as a function of reinforcer "hold": from probability learning to concurrent reinforcement

Abstract

Two procedures commonly used to study choice are concurrent reinforcement and probability learning. Under concurrent-reinforcement procedures, once a reinforcer is scheduled, it remains available indefinitely until collected. Therefore reinforcement becomes increasingly likely with passage of time or responses on other operanda. Under probability learning, reinforcer probabilities are constant and independent of passage of time or responses. Therefore a particular reinforcer is gained or not, on the basis of a single response, and potential reinforcers are not retained, as when betting at a roulette wheel. In the "real" world, continued availability of reinforcers often lies between these two extremes, with potential reinforcers being lost owing to competition, maturation, decay, and random scatter. The authors parametrically manipulated the likelihood of continued reinforcer availability, defined as hold, and examined the effects on pigeons' choices. Choices varied as power functions of obtained reinforcers under all values of hold. Stochastic models provided generally good descriptions of choice emissions with deviations from stochasticity systematically related to hold. Thus, a single set of principles accounted for choices across hold values that represent a wide range of real-world conditions.

Figure 1

Proportions of responses (solid circles and connecting lines) on left (L), middle (M), and right (R) keys for each of 6 pigeons (S1 through S6) during each session of Experiment 1. Also depicted are proportions of obtained reinforcers (open circles) and programmed reinforcer set-up probability (gray lines). These programmed probabilities, which were the same for all pigeons, are also written on the S1 graph for each phase.

Figure 2

Log of response ratios (left/middle, middle/right, and right/left) as a function of log obtained reinforcer ratios, these shown in the upper left, upper right, and lower left quadrants, respectively. Each point represents a pigeon’s performance during the last session of each phase. The lower right quadrant combines the data from the other three quadrants. The lines are the least-squares, best fitting functions.

Figure 3

Information contained in the distribution of response dyads (LL, LM, LR, ML …) as a function of the information value predicted if responses were stochastic. Data are from the last session in each phase of each of the experiments.

Figure 4

Mean run lengths on each key during the last session of each phase of Experiment 1 as a function of the proportion of responses to that key. Data from all of the pigeons are shown. The drawn line is the expected function if responses were stochastic.

Figure 5

The probability that a given response to key k would be reinforced as a function of the number of responses to other keys (i.e., responses since the last selection of key k, or n). The functions drawn are for different values of the hold parameter, shown to the right of the graph.

Figure 6

Proportions of responses (solid circles and connecting lines) on left, middle, and right keys averaged across the 6 pigeons during each session of Experiment 2. Also depicted are proportions of obtained reinforcers (open circles) and response proportions that would maximize reinforcement (gray lines). The hold values, which were the same for each of the keys, are shown on top of each graph.

Figure 7

Log ratios of response pairs (L/M, M/R, and R/L) as a function of the associated log ratios of obtained reinforcers for all pigeons in Experiment 2. Each point represents a pigeon’s performance during the last session of a phase, with the different hold values (phases) presented separately in five graphs, and the combination shown in the bottom right graph. The lines are the least-squares, best fitting functions.

Figure 8

Mean run lengths in Experiment 2 as functions of the proportions of L, M, and R responses. The separate graphs show performances under the different hold conditions. Data from all pigeons are combined on each graph. The drawn lines are the expected functions if responses were stochastic.

Figure 9

Ratio of the average pigeon switch proportions to expected switch proportions expected from a stochastic process as a function of hold values. Error bars represent standard errors.

Figure 10

Proportions of responses (solid circles and connecting lines) on left, middle, and right keys averaged across the 5 pigeons during each session of Experiment 3. Also depicted are proportions of obtained reinforcers (open circles) and response proportions that would maximize reinforcement (gray horizontal lines). The background white areas indicate hold values of 1.0 and the background gray areas indicate hold = 0.0.

Figure 11

Proportions of responses on left, middle, and right keys for individual subjects in Phase 5 (hold = 1.0) and Phase 6 (hold = 0.0) of Experiment 3. Each symbol represents 1 subject across the three graphs.

Figure 12

Log ratios of response pairs (L/M, M/R, and R/L) as a function of the associated log ratios of obtained reinforcers for all pigeons in Experiments 3 and 4. Each point represents a pigeon’s performance during the last session of a phase. The lines are the least-squares, best fitting functions.

Figure 13

Mean run lengths in Experiments 3 and 4 as functions of the proportions of responses. The drawn lines are the expected function if responses were stochastic.

Figure 14

Ratios of pigeon switch proportions to expected switch proportions from a stochastic process as a function of hold values in Experiments 3 and 4. Averages across the 5 pigeons are shown. Error bars represent standard errors.

Figure A1

Chart showing the probability of reinforcement at response n+1. Each circle represents a state of the contingency: Those marked 1 indicate that reinforcement is available, and those marked 0 indicate not available. For each state of the contingency, the downward leading arrows indicate the probability of the contingency moving to another state (the outgoing arrows always sum to 1.0). For details, see the text.

Figure B1

Flow chart of reinforcement contingencies in Experiments 2, 3, and 4.