Adapting to the algorithm: how accuracy comparisons promote the use of a decision aid

Abstract

In three experiments, we sought to understand when and why people use an algorithm decision aid. Distinct from recent approaches, we explicitly enumerate the algorithm's accuracy while also providing summary feedback and training that allowed participants to assess their own skills. Our results highlight that such direct performance comparisons between the algorithm and the individual encourages a strategy of selective reliance on the decision aid; individuals ignored the algorithm when the task was easier and relied on the algorithm when the task was harder. Our systematic investigation of summary feedback, training experience, and strategy hint manipulations shows that further opportunities to learn about the algorithm encourage not only increased reliance on the algorithm but also engagement in experimentation and verification of its recommendations. Together, our findings emphasize the decision-maker's capacity to learn about the algorithm providing insights for how we can improve the use of decision aids.

Keywords: Algorithm; Decision aid; Feedback.

Fig. 1

Examples of dot motion stimuli with the algorithm. In the task, a proportion of dots move along the 90°–270° axis coherently and participants judge the direction of dot movement along this axis as either left-motion or right-motion (shown in the orange arrows). Distractor dots move in straight lines but at different axes (shown in grey arrows). Note that the orange and grey arrows appear here only for illustrative purposes; they were never present for any participants in any experiment. Panel A shows the algorithm’s recommendation (left green arrow) above the stimulus. In Experiment 1a and 1b the recommendation appeared above the stimulus automatically. In Experiments 2 and 3 the algorithm appeared on screen as a green box (Panel C) unless the participant made a request for a recommendation. If requested, the algorithm loaded the recommendation during a one-second delay displaying a “loading circle” that revolved around the box (Panel B). All motion in the stimulus stopped during this loading time and then resumed once the recommendation was revealed (as in Panel A)

Fig. 2

Test block progression in Experiment 1a and Experiment 1b. Difficulty of the test stimuli was randomized within each block of trials. White proportion denotes the proportion of trials when the arrow/cancer cue appeared. In the control block, the cue never appeared. In the cued block, the cue appeared on a random half of the block. Though the figure may appear to show cue-absent trials appeared in the first half of the block, the actual order of cued and cue-absent trials was randomized within the block

Fig. 3

Proportion of correct responses during the test stage as a function of trial type and stimulus difficulty. Control trials, when the cue is never presented, are compared against cue-present trials in the cued block. Boxplots display the median and interquartile ranges with dots representing individual participants. Green intercept line represents the algorithm’s performance (i.e. 70% correct)

Fig. 4

Performance in the experiment for the training blocks (Panel A) and test stage (Panel B). Mean proportion correct is presented as a function of stimulus difficulty. Boxplots show median and interquartile ranges with dots representing outliers. Note that in training, stimuli were blocked by difficulty, i.e. participants underwent easier training block followed by harder training block in the easy-hard training condition. In test, difficulty was randomized for all conditions. Horizontal line in Panel B represents algorithm performance level of 70% correct. See online for colour version

Fig. 5

Panel A shows mean percentage of algorithm request trials as a function of condition and stimulus difficulty. Boxplot shows the median score and interquartile ranges. Panel B shows mean proportion correct with the recommendation on algorithm requested trials compared to unassisted “own decision” trials as a function of stimulus difficulty. See online for colour version

Fig. 6

Panel A presents the mean proportion correct as a function of difficulty. Intercept lines represent minimum incentive thresholds. An 80% average for easier images awarded $2.00. For the harder images, two out of six blocks of 65% correct awarded the additional $3.00 reward. Panel B presents the percentage of algorithm requests. Boxplots display median and interquartile ranges. As a reminder, the green Hint condition received a description of maximization and algorithm-performance comparison during block-feedback

Fig. 7

Panel A presents the correlation between algorithm requests and performance for the harder stimuli. Panel B is a histogram of the number of harder stimuli blocks in which participants reached the high-performance threshold of 65% correct. Darker shading indicates participants that received the additional payment, i.e. reached threshold in at least two blocks, lighter shading indicates no payment

Fig. 8

Algorithm agreement plotted as a function of whether the recommendation was correct (x-axis) and stimulus difficulty. Data grouped by condition from Experiments 2 and 3. Agreement measures the proportion of trials when the participant made the same response as recommended. Note that only trials in which the recommendation was requested is there agreement data. Boxplots display medians and interquartile ranges with dots displaying individual participant data. See online for colour version

Fig. 9

Algorithm requests as a function of initial experiences with the algorithm. X-axis plots the proportion of algorithm mistakes for easier stimuli in the first 20 trials of the experiment. Y-axis plots the subsequent proportion of algorithm requests for the harder stimuli. Diagonal dotted line illustrates the expected relationship that initially seeing more algorithm mistakes would result in less reliance on the algorithm. Coloured horizontal lines show line of best fit for each experiment. Proportion is plotted on the x-axis to account for the different number of trials across participants