A multivariate model for successful publication of intensive care medicine randomized controlled trials in the highest impact factor journals: the SCOTI score

Abstract

Background: Critical care randomized controlled trials (RCTs) are often published in high-impact journals, whether general journals [the New England Journal of Medicine (NEJM), The Lancet, the Journal of the American Medical Association (JAMA)] or critical care journals [Intensive Care Medicine (ICM), the American Journal of Respiratory and Critical Care Medicine (AJRCCM), Critical Care Medicine (CCM)]. As rejection occurs in up to 97% of cases, it might be appropriate to assess pre-submission probability of being published. The objective of this study was to develop and internally validate a simplified score predicting whether an ongoing trial stands a chance of being published in high-impact general journals.

Methods: A cohort of critical care RCTs published between 1999 and 2018 in the three highest impact medical journals (NEJM, The Lancet, JAMA) or the three highest impact critical care journals (ICM, AJRCCM, CCM) was split into two samples (derivation cohort, validation cohort) to develop and internally validate the simplified score. Primary outcome was journal of publication assessed as high-impact general journal (NEJM, The Lancet, JAMA) or critical care journal (ICM, AJRCCM, CCM).

Results: A total of 968 critical care RCTs were included in the predictive cohort and split into a derivation cohort (n = 510) and a validation cohort (n = 458). In the derivation cohort, the sample size (P value < 0.001), the number of centers involved (P value = 0.01), mortality as primary outcome (P value = 0.002) or a composite item including mortality as primary outcome (P value = 0.004), and topic [ventilation (P value < 0.001) or miscellaneous (P value < 0.001)] were independent factors predictive of publication in high-impact general journals, compared to high-impact critical care journals. The SCOTI score (Sample size, Centers, Outcome, Topic, and International score) was developed with an area under the ROC curve of 0.84 (95% Confidence Interval, 0.80-0.88) in validation by split sample.

Conclusions: The SCOTI score, developed and validated by split sample, accurately predicts the chances of a critical care RCT being published in high-impact general journals, compared to high-impact critical care journals.

Keywords: Critical care; Critically ill patients; Intensive care medicine; Intensive care unit; Precision medicine.

Fig. 1

Flow chart of the study

Fig. 2

Receiver Operating Characteristic curves and Areas Under the Curve to predict publication in high-impact general journals, compared to high-impact critical care journals: a final model in development cohort. b Final model in validation cohort. c SCOTI score in development cohort. d SCOTI score in validation cohort. AUC: Area under the curve, 95%CI: Confidence Interval at 95%

Fig. 3

SCOTI score. a SCOTI score calculation worksheet. SCOTI score: sample size, centers, outcome, topic, and international score. b Probability of publication in high-impact general journals compared to high-impact critical care journals according to the SCOTI score: individual prediction. The plot shows predicted (orange curve) and observed (blue dots) probabilities of publication in high-impact general journals according to the SCOTI score. c Probability of publication in high-impact medical journals compared to high-impact critical care journals according to the SCOTI score: three categories. The plot shows observed probabilities of publication in high-impact general journals according to three groups of SCOTI score: “Low probability of success”, “Intermediate probability of success” and “High probability of success”