How one block of trials influences the next: persistent effects of disease prevalence and feedback on decisions about images of skin lesions in a large online study

Abstract

Using an online, medical image labeling app, 803 individuals rated images of skin lesions as either "melanoma" (skin cancer) or "nevus" (a skin mole). Each block consisted of 80 images. Blocks could have high (50%) or low (20%) target prevalence and could provide full, accurate feedback or no feedback. As in prior work, with feedback, decision criteria were more conservative at low prevalence than at high prevalence and resulted in more miss errors. Without feedback, this low prevalence effect was reversed (albeit, not significantly). Participants could participate in up to four different conditions a day on each of 6 days. Our main interest was in the effect of Block N on Block N + 1. Low prevalence with feedback made participants more conservative on a subsequent block. High prevalence with feedback made participants more liberal on a subsequent block. Conditions with no feedback had no significant impact on the subsequent block. The delay between Blocks 1 and 2 had no significant effect. The effect on the second half of Block 2 was just as large as on the first half. Medical expertise (over the range available in the study) had no impact on these effects, though medical students were better at the task than other groups. Overall, these seem to be robust effects where feedback may be 'teaching' participants how to respond in the future. This might have application in, for example, training or re-training situations.

Fig. 1

Sample stimulus displays. Left: participants are asked if a lesion is a melanoma (cancer) or a nevus (benign ‘mole’). Right: after response, if this were a feedback condition, red or green feedback would inform the participant of the correctness of the answer. In no feedback conditions, neutral feedback indicated that the response had been registered

Fig. 2

D′ and criterion as a function of condition. Each dot represents one of 2080 blocks of data. Black lines show mean and ± 95% CI of the mean

Fig. 3

Median response time as a function of block type. RTs < 11 s were analyzed. A RT distributions with the graph truncated at 3 s for display purposes. Black lines show the mean. B Means of the median RTs on a much finer scale. Error bars show ± 95% confidence intervals

Fig. 4

Change in d′ on block 2 as a function of the nature of block 1. p values show results for simple t-tests, testing against the null hypothesis that there is no change in d′. Black lines show means ± 95% confidence intervals

Fig. 5

Change in criterion on block 2 as a function of the nature of block 1. P values show results for simple t-tests, testing against the null hypothesis that there is no change in criterion. Black lines show means ± 95% confidence intervals

Fig. 6

Change in criterion as a function of time between the start of block 1 and start of block 2. 1440 min = 1 day. Colored lines are best-fit linear regressions for each of 16 pairs of block types. A All data, B Delay < 1 day, C Delay < 2 h, D Delay < 15 min. Repeat pairs (e.g., Low Feedback → Low Feedback) do not occur for shorter delays (C, D)

Fig. 7

Change in criterion on block 2 as a function of the nature of block 1. Data restricted to pairs separated by more than 2 h

Fig. 8

Mean change in criterion for the first and second halves of a block of trials

Fig. 9

Effects of expertise category on d′ and crit, regardless of prevalence and/or feedback

Fig. 10

Hypothetical curves showing the results of the feedback and no feedback conditions as two points on continuous functions