Effect of using reporting guidelines during peer review on quality of final manuscripts submitted to a biomedical journal: masked randomised trial

Abstract

Objective: To investigate the effect of an additional review based on reporting guidelines such as STROBE and CONSORT on quality of manuscripts.

Design: Masked randomised trial. Population Original research manuscripts submitted to the Medicina Clínica journal from May 2008 to April 2009 and considered suitable for publication.

Control group: conventional peer reviews alone. Intervention group: conventional review plus an additional review looking for missing items from reporting guidelines. Outcomes Manuscript quality, assessed with a 5 point Likert scale (primary: overall quality; secondary: average quality of specific items in paper). Main analysis compared groups as allocated, after adjustment for baseline factors (analysis of covariance); sensitivity analysis compared groups as reviewed. Adherence to reviewer suggestions assessed with Likert scale.

Results: Of 126 consecutive papers receiving conventional review, 34 were not suitable for publication. The remaining 92 papers were allocated to receive conventional reviews alone (n=41) or additional reviews (n=51). Four papers assigned to the conventional review group deviated from protocol; they received an additional review based on reporting guidelines. We saw an improvement in manuscript quality in favour of the additional review group (comparison as allocated, 0.25, 95% confidence interval -0.05 to 0.54; as reviewed, 0.33, 0.03 to 0.63). More papers with additional reviews than with conventional reviews alone improved from baseline (22 (43%) v eight (20%), difference 23.6% (3.2% to 44.0%), number needed to treat 4.2 (from 2.3 to 31.2), relative risk 2.21 (1.10 to 4.44)). Authors in the additional review group adhered more to suggestions from conventional reviews than to those from additional reviews (average increase 0.43 Likert points (0.19 to 0.67)).

Conclusions: Additional reviews based on reporting guidelines improve manuscript quality, although the observed effect was smaller than hypothesised and not definitively demonstrated. Authors adhere more to suggestions from conventional reviews than to those from additional reviews, showing difficulties in adhering to high methodological standards at the latest research phases. To boost paper quality and impact, authors should be aware of future requirements of reporting guidelines at the very beginning of their study. Trial registration and protocol Although registries do not include trials of peer review, the protocol design was submitted to sponsored research projects (Instituto de Salud Carlos III, PI081903).

Fig 1 Study design and manuscript flow. *Additional reviews and measurement of minimisation variables were undertaken during the standard peer review process, but this information was concealed until the later editorial stages

Fig 2 Goodman quality scores at baseline. Bars contain proportion of scores from 1 (dark shade) to 5 (light shade), with cumulative percentages shown in the bottom scale. Gradation colour for the average quality was consequently adapted (break points are 2.5, 3.0, 3.5, and 4.0). Dots represent total mean for each specific item

Fig 3 Effect of additional reviews on overall and average quality in “as allocated” and “as reviewed” populations, and primary analysis stratified by study type

Fig 4 Baseline and final Goodman quality scores in allocated groups. Numbers after the plus signs indicate the four manuscripts with protocol deviations

Fig 5 Average author adherence to repeated reviewer suggestions based on 5 point Likert scale (1=minimum, 5=maximum). Rectangles represent the 51 papers from the additional review group, with paired suggestions both from conventional reviews (shown as the horizontal lines on each rectangle) and additional reviews (shown as the vertical lines of each rectangle). The side length of rectangles represents the amount of information from any type of review (square root of the number of suggestions per manuscript) and the rectangle area represents each paper’s overall information. A rectangle above the diagonal line indicates that a paper adhered more to the conventional review than to the additional review. For example, the asterisked rectangle corresponds to a manuscript receiving 14 suggestions (proportional to the square of the vertical sides) from the additional review with a 3.21 average level of adherence, and two suggestions (the square of the horizontal sides) from the conventional review with a mean adherence score of 5. Lines in the external margin represent papers from the conventional review group, and lines on the internal margin represent papers from the additional review group that received both conventional (lines along the vertical axis) and additional (lines along the horizontal axis) reviews; lines are repeated here to assist between group comparison.