Modelling Influence and Opinion Evolution in Online Collective Behaviour

Abstract

Opinion evolution and judgment revision are mediated through social influence. Based on a large crowdsourced in vitro experiment (n = 861), it is shown how a consensus model can be used to predict opinion evolution in online collective behaviour. It is the first time the predictive power of a quantitative model of opinion dynamics is tested against a real dataset. Unlike previous research on the topic, the model was validated on data which did not serve to calibrate it. This avoids to favor more complex models over more simple ones and prevents overfitting. The model is parametrized by the influenceability of each individual, a factor representing to what extent individuals incorporate external judgments. The prediction accuracy depends on prior knowledge on the participants' past behaviour. Several situations reflecting data availability are compared. When the data is scarce, the data from previous participants is used to predict how a new participant will behave. Judgment revision includes unpredictable variations which limit the potential for prediction. A first measure of unpredictability is proposed. The measure is based on a specific control experiment. More than two thirds of the prediction errors are found to occur due to unpredictability of the human judgment revision process rather than to model imperfection.

Fig 1. Influenceability over rounds and games.

(A) Gauging game, (B) Counting game: distributions of the α_i(1) influenceability after the first round and α_i(2) influenceability after the second round for the time-varying influenceability model (1). The colormap corresponds to the average prediction RMSE of participants in each bin. For visualization purposes, one value at α_i(1) = −1.5 has been removed from the histogram in the gauging game, round 1. (There was only one individual with α_i(1) = −1.5. For this particular individual, the linear relationship between x_i(2) − x_i(1) and $\overline{x}\left(1\right)-x_i\left(1\right)$ is not significant (p-val = 0.38), so that the coefficient α_i(1) = −1.5 should not be interpreted. For the rest of the participants, the level of trust can be read from the color scale. The color of a bar corresponds to the average prediction error made for the participants with α_i values falling within the bar range). (C) Cumulated distributions of α_i(1) and α_i(2) for each type of games.

Fig 2. Root mean square error (RMSE) of the predictions for the final round.

The RMSEs are obtained from crossvalidation. (A) gauging game, (B) counting game. In (B), the RMSE has been scaled by a factor of 5 to be comparable to the (A) plot. The bar chart displays crossvalidation errors for models that does not depend on the training size. Top blue horizontal line corresponds to the null model of constant opinion. The middle horizontal green line corresponds to fitting using the same typical couple of influenceability for the whole population. The bottom horizontal red line corresponds to the prediction error due to intrinsic variations in judgment revision. The decreasing black curve (triangle dots) corresponds to fitting with the individual influenceability method. The slightly decreasing orange curve (round dots) corresponds to fitting choosing among 2 typical couples of influenceability. All RMSE were obtained on validation games. The error bars provide 95% confidence intervals for the RMSEs.

Fig 3. Predictive power of the consensus model when the number of typical couples of influenceability used for model calibration varies.

Root mean square error (RMSE) plotted for training set size from 1 to 15 games. (A) gauging game, (B) counting game. In (B), RMSE has been scaled by a factor of 5 to be comparable to the (A) plot.

Fig 4. Distance of participants’ judgment to mean judgment for rounds 1, 2 and 3 in the gauging game.

Is displayed only the data coming from games where for all 3 rounds at least 5 out of 6 participants provided a judgment. This ensures that the judgment mean and standard deviation can be compared over rounds.

Fig 5

(A) gauging game; (B) counting game. From left to right: (Boxplot 1, 3 and 5, in blue) Root mean square distance from individual opinions to truth (E_i(r)); (Boxplot 2, 4 and 6 in red) Root mean square distance from mean opinion to truth. Values shown for the counting game have been scaled by a factor of 5.

Fig 6. Illustration of the shift applied to the synthetic judgment in order to preserve the distances between initial judgments.

The shift made by the participant in the first round was synthetically applied to all other initial judgments. The intrinsic variation in judgment revision is the distance between second round judgments, reduced by the shift in initial judgments. The dotted cross is not a judgment and is only displayed to show how the second round intrinsic variation is computed using the initial shift. All shifts are equal. The same shift is also applied to all other second round judgments, and used to compute final round intrinsic variations.

Fig 7. Filters for participants’ trustworthiness.

(A),(C) gauging game, (B),(D) counting game. (A,B): Histograms of the number of full games played by participants. (C,D): Histograms of the correlations between judgments and true values for each participant. Rejected participants are displayed in red while kept participants are in blue, to the right of the black arrow.

Fig 8. Root mean square intrinsic variation as a function of the λ parameters.

(A), (B): the second round judgments, as given in Eq (3). (C), (D): the third round judgments, as given in Eq (5). (A), (C) gauging game, (B), (D) counting game.

Fig 9. Root mean square error (RMSE) of the predictions for the second round.

The RMSEs are obtained from crossvalidation. (A) gauging game, (B) counting game. In (B), the RMSE has been scaled by a factor of 5 to be comparable to the (A) plot. The bar chart displays crossvalidation errors for models that does not depend on the training set size. Top blue horizontal line corresponds to the null model of constant opinion. The middle horizontal green line correspond to fitting using the same typical couple of influenceability for the whole population. The bottom horizontal red line correspond to the prediction error due to intrinsic variations in judgment revision. The decreasing black curves (triangle dots) correspond to fitting with the individual influenceability method. The slightly decreasing orange curves (round dots) correspond to fitting choosing among 2 typical couples of influenceability. All RMSE were obtained on validation games.