Current Incentives for Scientists Lead to Underpowered Studies with Erroneous Conclusions

Abstract

We can regard the wider incentive structures that operate across science, such as the priority given to novel findings, as an ecosystem within which scientists strive to maximise their fitness (i.e., publication record and career success). Here, we develop an optimality model that predicts the most rational research strategy, in terms of the proportion of research effort spent on seeking novel results rather than on confirmatory studies, and the amount of research effort per exploratory study. We show that, for parameter values derived from the scientific literature, researchers acting to maximise their fitness should spend most of their effort seeking novel results and conduct small studies that have only 10%-40% statistical power. As a result, half of the studies they publish will report erroneous conclusions. Current incentive structures are in conflict with maximising the scientific value of research; we suggest ways that the scientific ecosystem could be improved.

Fig 1. Fitness landscape for an individual researcher.

An individual researcher is able to choose the parameters θ (y-axis) and S_E; the x-axis shows the resultant power of exploratory studies, W_E. White indicates high fitness, black low fitness. For small values of S_E, few papers are accepted, while for high values of S_E, few studies are carried out. For low values of θ, few novel studies are carried out. (A) γ = 0.09, ϕ = 0.9. The optimal strategy that maximises individual fitness is therefore to carry out many small exploratory studies with a power of around 15%. (B) γ = 0.055, ϕ = 0.55. A mixture of exploratory and confirmatory work should be carried out with slightly higher power (20%).

Fig 2. Effect of varying the weighting given to published exploratory studies (γ).

Parameter γ reflects the relative importance of published exploratory studies. The lines show predictions for two values of the probability that an effect is real (f_E) and two values of the effect sizes r_C and r_E (solid: f_E = 0.2, r_C = r_E = 0.21; dotted: f_E = 0.3, r_C = r_E = 0.21; dashed: f_E = 0.2, r_C = r_E = 0.32). The panels show (A) the optimal proportion of total sampling to spend on exploratory studies θ*, (B) the optimal sample size of exploratory studies S_E*, (C) the resultant total number of published studies N_E + N_C, (D) the proportion of published studies that are confirmatory N_C / (N_E + N_C), (E) the statistical power of exploratory studies W_E, and (F) the proportion of published studies that draw incorrect conclusions (P_F). Other values: S_C = 120, T = 2,000, k = 20, α = 0.05, σ² = 1, m = 3, and ϕ = 0.8. The chosen values for r_C = r_E reflect data reported by Richard and colleagues [14], where a correlation coefficient mode of 0.09 and a mean of 0.21 were observed. These values are in the middle of the range of effect sizes observed in meta-analyses across a number of biomedical research domains (range r ~ 0.15 to 0.50) [13].

Fig 3. Effect of γ and ϕ on a hypothetical measure of the total scientific value of research (V_S).

The figure shows the product of the number of published confirmatory studies, the number of published exploratory studies, and the proportion of published studies that are correct (red = high, blue = low). This measure is calculated for when all researchers are following the rational strategy given the values of γ and ϕ. The current emphasis on a small number of publications that report novel findings is characterised by high γ and high ϕ (top right). To improve scientific output according to this measure, we could reduce ϕ (i.e., make more published studies count for researchers’ careers) or reduce γ (i.e., reduce weighting of published exploratory studies). Interestingly, the ridge is flat, so any point along it has equal fitness. Therefore, a pragmatic compromise would be to reduce both γ and ϕ by a lesser amount. The panels show the V_T for two values of the dependence of acceptance on sample size m and the Type I error rate α: (A) α = 0.05, m = 3, colour range: 2.0–3.18; (B) α = 0.05, m = 6, colour range: 2.0–2.82; (C) α = 0.03, m = 6, colour range: 2.0–2.065. Other values: S_C = 120, T = 2000, k = 20, f_E = 0.2, r_C = r_E = 0.21, and σ² = 1.

Fig 4. Effect of editorial stringency on total scientific output for current incentive structures.

The figure shows the proportion of published findings that are correct 1-P_F (A, B), the total number of published studies N_C + N_C (C, D), and the total scientific value of research V_T (E, F). We varied the following parameters: the probability of a Type I error α (A, C, E), and the dependence of acceptance on sample size m (B, D, F). The lines show predictions for two values of the probability that an effect is real (f_E) and two values of the effect size (solid: f_E = 0.2, r_C = r_E = 0.21; dotted: f_E = 0.3, r_C = r_E = 0.21; and dashed: f_E = 0.2, r_C = r_E = 0.32). Other values: N_C = 120, T = 2,000, k = 20, α = 0.05, σ² = 1, m = 3, ϕ = 0.9, and γ = 0.09. The steps occur where there is discontinuity in the effect of α on S_E*.

Fig 5. Effect of editorial stringency on total scientific output for ideal incentive structures.

The figure shows the proportion of published studies that are correct 1 − P_T (A, B), the total number of published studies N_C + N_C (C, D), and the total scientific value of research V_S (E, F). We varied the following parameters: the probability of a Type I error α (A, C, E), and the dependence of acceptance on sample size m (B, D, F). The lines show predictions for two values of the probability that an effect is real (f_E) and two values of the effect size (solid: f_E = 0.2, r_C = r_E = 0.21; dotted: f_E = 0.3, r_C = r_E = 0.21; dashed: f_E = 0.2, r_C = r_E = 0.32). Other values: N_C = 120, T = 2,000, k = 20, α = 0.05, σ² = 1, m = 3, ϕ = 0.55, and γ = 0.055.