Wisdom of crowds benefits perceptual decision making across difficulty levels

Abstract

Decades of research on collective decision making has claimed that aggregated judgment of multiple individuals is more accurate than expert individual judgement. A longstanding problem in this regard has been to determine how decisions of individuals can be combined to form intelligent group decisions. Our study consisted of a random target detection task in natural scenes, where human subjects (18 subjects, 7 female) detected the presence or absence of a random target as indicated by the cue word displayed prior to stimulus display. Concurrently the neural activities (EEG signals) were recorded. A separate behavioural experiment was performed by different subjects (20 subjects, 11 female) on the same set of images to categorize the tasks according to their difficulty levels. We demonstrate that the weighted average of individual decision confidence/neural decision variables produces significantly better performance than the frequently used majority pooling algorithm. Further, the classification error rates from individual judgement were found to increase with increasing task difficulty. This error could be significantly reduced upon combining the individual decisions using group aggregation rules. Using statistical tests, we show that combining all available participants is unnecessary to achieve minimum classification error rate. We also try to explore if group aggregation benefits depend on the correlation between the individual judgements of the group and our results seem to suggest that reduced inter-subject correlation can improve collective decision making for a fixed difficulty level.

Figure 1

Experimental Protocol and Participant Behaviour. (A) Experimental Paradigm and example of Target Present and Target Absent natural scene images. (B) Behavioural performance of the seventeen participants. (C) Correlation between difficulty of the trials and mean individual confidence of the participants. Individual confidence decreases with task difficulty.

Figure 2

Univariate neural data analysis. (A) Topographic plot of Target Present—Target Absent difference wave. A dominant parietal activity is observed 300 ms onwards. The color-bar represents the magnitude of the difference wave in $\mu V$. (B) ERP waveforms corresponding to Target Present and Target Absent trials. Mean taken across 4 electrodes—P3, Pz, P4 and POz. Shaded area indicate ± SEM. The position of the 4 electrodes are marked using black dots in the topoplots of (A).

Figure 3

Group Aggregation. (A,B) Performance benefits achieved using the commonly used majority pooling algorithm and the weighted average algorithm across group sizes from behavioural (A) and neural (B) data. The proportion of correct trials is consistently more upon combining the individual decisions using the weighted average rule. Error bars indicate ± SD. (C) Control study to check that the the performance benefits achieved are not due to addition of more electrodes but due to multiple brains. The PC increases when doubling the number of heads relative to doubling the number of electrodes. ‘Ind30’ denotes single brain with 30 electrodes, ‘Ind59’ denotes single brain with all 59 electrodes and ‘2heads’ denotes double brain with 30 electrodes each. Error bars denote ± SEM.

Figure 4

Relationship between Behavioural and Neural Decision Variables (A)–(E) Relationship between the behavioural and neural decision variables from the training data across difficulty levels 1 to 5. (F) Partial correlation between the behavioural and neural decision variables from the training data after controlling for effect of difficulty. Colour bars represent the frequency of observations.

Figure 5

Group Aggregation and Task Difficulty. (A) and (C) depict the classification error rate across group sizes and difficulty levels for behavioural and neural data. Classification error rate increases with increasing task difficulty. This error is significantly reduced upon combining the individual decisions using the group aggregation rule. The colour bars indicate the magnitude of classification error rate. (B) and (D) show percentages of the benefit of group decision making (maximum group size vs individual) as a function of difficulty level for behavioural and neural data, respectively. Benefits reduce with increasing difficulty. Error bars represent ± SD.

Figure 6

(A) and (B) Group-ISC as a function of task difficulty in behavioural and neural data, respectively. In both cases, Group-ISC is lower at difficult trials. Error bars denote ± SEM. (C) Neural timeline of Group-ISC at different difficulty levels. A sudden peak in the Group-ISC is noticed in the time window 200-240 ms after stimulus onset. The value is reduced with increasing task difficulty. In figures (A–C) the Group-ISC is maximum at the 2nd difficulty level and minimum at 5th difficulty level. (D) Neural activity in multiple brains across time at difficulty levels 1 and 5 for individual and group decision (Group Size 13).