Beyond Binary Decisions: Evaluating the Effects of AI Error Type on Trust and Performance in AI-Assisted Tasks

Abstract

ObjectiveWe investigated how various error patterns from an AI aid in the nonbinary decision scenario influence human operators' trust in the AI system and their task performance.BackgroundExisting research on trust in automation/autonomy predominantly uses the signal detection theory (SDT) to model autonomy performance. The SDT classifies the world into binary states and hence oversimplifies the interaction observed in real-world scenarios. Allowing multi-class classification of the world reveals intriguing error patterns previously unexplored in prior literature.MethodThirty-five participants completed 60 trials of a simulated mental rotation task assisted by an AI with 70-80% reliability. Participants' trust in and dependence on the AI system and their performance were measured. By combining participants' initial performance and the AI aid's performance, five distinct patterns emerged. Mixed-effects models were built to examine the effects of different patterns on trust adjustment, performance, and reaction time.ResultsVarying error patterns from AI impacted performance, reaction times, and trust. Some AI errors provided false reassurance, misleading operators into believing their incorrect decisions were correct, worsening performance and trust. Paradoxically, some AI errors prompted safety checks and verifications, which, despite causing a moderate decrease in trust, ultimately enhanced overall performance.ConclusionThe findings demonstrate that the types of errors made by an AI system significantly affect human trust and performance, emphasizing the need to model the complicated human-AI interaction in real life.ApplicationThese insights can guide the development of AI systems that classify the state of the world into multiple classes, enabling the operators to make more informed and accurate decisions based on feedback.

Keywords: human–AI interaction; human–automation interaction; human–autonomy interaction; multi-class classification; trust dynamics.

Figure 1.

Difference in the performance patterns when choices are binary or non-binary. In the binary choice (a), the Wrong-Incorrect pattern always matches the reference. However, in the non-binary choice (b), the Wrong-Incorrect pattern could be categorized into two, based on whether incorrect predictions match the reference (prescribed pill).

Figure 1.

Difference in the performance patterns when choices are binary or non-binary. In the binary choice (a), the Wrong-Incorrect pattern always matches the reference. However, in the non-binary choice (b), the Wrong-Incorrect pattern could be categorized into two, based on whether incorrect predictions match the reference (prescribed pill).

Figure 2.

Signal Detection Theory (SDT)

Figure 3.

Illustration of stimuli development. After creating the (a) reference images, one (b) mirrored image, fourteen 45-degree (c) horizontal axis rotated figures, and fourteen 45-degree (d) vertical axis rotated figures were created for each reference image.

Figure 4.

For the beta test, participants were asked to determine whether the image pairs are the same items in different rotations (by clicking “Yes, they are the same item”) or different items (by clicking “No, they are different”) within 10 seconds.

Figure 5.

Flowchart of the experiment

Figure 6.

Participants were asked to make their initial answer choice within 15 seconds.

Figure 6.

Participants were asked to make their initial answer choice within 15 seconds.

Figure 7.

Participants rated their confidence using a visual analog scale.

Figure 8.

Participants were presented with the AI system’s recognition and chose between sticking with or rejecting their initial choice within 10 seconds. If no selection was made, they were skipped to the next trial.

Figure 8.

Participants were presented with the AI system’s recognition and chose between sticking with or rejecting their initial choice within 10 seconds. If no selection was made, they were skipped to the next trial.

Figure 9.

Participants were presented with the performance feedback page, showing their performance on the initial and final answer choices and the validity of the AI’s prediction.

Figure 9.

Participants were presented with the performance feedback page, showing their performance on the initial and final answer choices and the validity of the AI’s prediction.

Figure 10.

Participants rated their trust and perceived reliability using a visual analog scale.

Figure 11.

Trust adjustment by patterns of performance. The error bars represent +/− 2 standard errors.

Figure 12.

Performance by patterns of performance. The error bars represent +/− 2 standard errors.

Figure 13.

Reaction Time by patterns of performance. The error bars represent +/− 2 standard errors.

Figure 13.

Reaction Time by patterns of performance. The error bars represent +/− 2 standard errors.

Figure 14.

Autocorrelation of trust as a function of time separation. The error bars represent +/− 2 standard errors.

Figure 15.

Trust adjustment by patterns of performance. The error bars represent +/− 2 standard errors.