Statistical reviewers improve reporting in biomedical articles: a randomized trial

Abstract

Background: Although peer review is widely considered to be the most credible way of selecting manuscripts and improving the quality of accepted papers in scientific journals, there is little evidence to support its use. Our aim was to estimate the effects on manuscript quality of either adding a statistical peer reviewer or suggesting the use of checklists such as CONSORT or STARD to clinical reviewers or both.

Methodology and principal findings: Interventions were defined as 1) the addition of a statistical reviewer to the clinical peer review process, and 2) suggesting reporting guidelines to reviewers; with "no statistical expert" and "no checklist" as controls. The two interventions were crossed in a 2x2 balanced factorial design including original research articles consecutively selected, between May 2004 and March 2005, by the Medicina Clinica (Barc) editorial committee. We randomized manuscripts to minimize differences in terms of baseline quality and type of study (intervention, longitudinal, cross-sectional, others). Sample-size calculations indicated that 100 papers provide an 80% power to test a 55% standardized difference. We specified the main outcome as the increment in quality of papers as measured on the Goodman Scale. Two blinded evaluators rated the quality of manuscripts at initial submission and final post peer review version. Of the 327 manuscripts submitted to the journal, 131 were accepted for further review, and 129 were randomized. Of those, 14 that were lost to follow-up showed no differences in initial quality to the followed-up papers. Hence, 115 were included in the main analysis, with 16 rejected for publication after peer review. 21 (18.3%) of the 115 included papers were interventions, 46 (40.0%) were longitudinal designs, 28 (24.3%) cross-sectional and 20 (17.4%) others. The 16 (13.9%) rejected papers had a significantly lower initial score on the overall Goodman scale than accepted papers (difference 15.0, 95% CI: 4.6-24.4). The effect of suggesting a guideline to the reviewers had no effect on change in overall quality as measured by the Goodman scale (0.9, 95% CI: -0.3-+2.1). The estimated effect of adding a statistical reviewer was 5.5 (95% CI: 4.3-6.7), showing a significant improvement in quality.

Conclusions and significance: This prospective randomized study shows the positive effect of adding a statistical reviewer to the field-expert peers in improving manuscript quality. We did not find a statistically significant positive effect by suggesting reviewers use reporting guidelines.

Figure 1. Flow chart of the review process among the four randomization groups.

From 131 selected papers for possible publication, 129 were randomized and allocated into four comparison groups in a 2×2 factorial design to evaluate the effects of adding a statistical expert as a reviewer and of suggesting the use of checklists. Main analyses are based on the 115 papers which completed the review process.

Figure 2. Mean quality changes, with uncertainty confidence intervals, from initial version to final version of the full Goodman scale.

The overall quality is computed by the addition of the scores obtained in each one of the 36 items of the scale (first item is a summarizing measure of quality and thus, not included). The two groups without a statistical reviewer had a final-initial difference of 4.5 and 4.7 points, equivalent to a final-initial change in one item from very poor to excellent. Indeed, the two groups with a methodological referee changed 9.1 and 11.3, more than a complete reversal in the evaluation of two items. Thus, the effect size of adding a statistical reviewer is computed as the weighted average of the changes in groups CS+CSK minus changes in C+CK.

Figure 3. Intervention effects on overall Goodman Scale.

Solid boxes indicate statistically significant effects.

Figure 4. Estimated effect of statistical review over the 36 Goodman items.

Solid boxes indicate statistically significant effects. ¹ Differences in improvement from baseline between groups with and without statistical reviewer

Figure 5. Estimated effect of checklist suggestion on the 36 Goodman items.

Solid boxes indicate statistically significant effects. ¹ Differences in improvement from baseline between groups with and without checklist suggestion