Do people experience cognitive biases while searching for information?

Abstract

Objective: To test whether individuals experience cognitive biases whilst searching using information retrieval systems. Biases investigated are anchoring, order, exposure and reinforcement.

Design: A retrospective analysis and a prospective experiment were conducted to investigate whether cognitive biases affect the way that documentary evidence is interpreted while searching online. The retrospective analysis was conducted on the search and decision behaviors of 75 clinicians (44 doctors, 31 nurses), answering questions for 8 clinical scenarios within 80 minutes in a controlled setting. The prospective study was conducted on 227 undergraduate students, who used the same search engine to answer two of six randomly assigned consumer health questions.

Measurements: Frequencies of correct answers pre- and post- search, and confidence in answers were collected. The impact of reading a document on the final decision was measured by the population likelihood ratio (LR) of the frequency of reading the document and the frequency of obtaining a correct answer. Documents with a LR > 1 were most likely to be associated with a correct answer, and those with a LR < 1 were most likely to be associated with an incorrect answer to a question. Agreement between a subject and the evidence they read was estimated by a concurrence rate, which measured the frequency that subjects' answers agreed with the likelihood ratios of a group of documents, normalized for document order, time exposure or reinforcement through repeated access. Serial position curves were plotted for the relationship between subjects' pre-search confidence, document order, the number of times and length of time a document was accessed, and concurrence with post-search answers. Chi-square analyses tested for the presence of biases, and the Kolmogorov-Smirnov test checked for equality of distribution of evidence in the comparison populations.

Results: A person's prior belief (anchoring) has a significant impact on their post-search answer (retrospective: P < 0.001; prospective: P < 0.001). Documents accessed at different positions in a search session (order effect [retrospective: P = 0.76; prospective: P = 0.026]), and documents processed for different lengths of time (exposure effect [retrospective: P = 0.27; prospective: P = 0.0081]) also influenced decision post-search more than expected in the prospective experiment but not in the retrospective analysis. Reinforcement through repeated exposure to a document did not yield statistical differences in decision outcome post-search (retrospective: P = 0.31; prospective: P = 0.81).

Conclusion: People may experience anchoring, exposure and order biases while searching for information, and these biases may influence the quality of decision making during and after the use of information retrieval systems.

Figure 1

Anchoring effect: the null hypothesis (H0) and the alternative hypothesis (H1).

Figure 2

Confidence in anchoring effect: the null hypothesis (H0) and the alternative hypothesis (H1).

Figure 3

Order effect: the null hypothesis (H0) and the alternative hypothesis (H1).

Figure 4

Exposure effect: the null hypothesis (H0) and the alternative hypothesis (H1).

Figure 5

Reinforcement effect: the null hypothesis (H0) and the alternative hypothesis (H1).

Figure 6

Data exclusion in retrospective analysis.

Figure 7

Data exclusion in prospective experiment.

Figure 8

Relationship between pre-search answer and post-search correctness (please refer to ▶ for frequency numbers).

Figure 9

Serial anchor curve: relationship between confidence in pre-search answer and retention of pre-search answer after searching.

Figure 10

Relationship between document access position and concurrence rate between post-search answer and document-suggested answer.

Figure 11

Relationship between document exposure level and concurrence rate between post-search answer and document-suggested answer.

Figure 12

Relationship between document access frequency and concurrence rate between post-search answer and document-suggested answer.