Influence of overstated abstract conclusions on clinicians: a web-based randomised controlled trial

Abstract

Objectives: To investigate whether overstatements in abstract conclusions influence primary care physicians' evaluations when they read reports of randomised controlled trials (RCTs) DESIGN: RCT setting: This study was a parallel-group randomised controlled survey, conducted online while masking the study hypothesis.

Participants: Volunteers were recruited from members of the Japan Primary Care Association in January 2017. We sent email invitations to 7040 primary care physicians. Among the 787 individuals who accessed the website, 622 were eligible and automatically randomised into 'without overstatement' (n=307) and 'with overstatement' (n=315) groups.

Interventions: We selected five abstracts from published RCTs with at least one non-significant primary outcome and overstatement in the abstract conclusion. To construct a version without overstatement, we rewrote the conclusion sections. The methods and results sections were standardised to provide the necessary information of primary outcome information when it was missing in the original abstract. Participants were randomly assigned to read an abstract either with or without overstatements and asked to evaluate the benefit of the intervention.

Outcome measures: The primary outcome was the participants' evaluation of the benefit of the intervention discussed in the abstract, on a scale from 0 to 10. A secondary outcome was the validity of the conclusion.

Results: There was no significant difference between the groups with respect to their evaluation of the benefit of the intervention (mean difference: 0.07, 95% CI -0.28 to 0.42, p=0.69). Participants in the 'without' group considered the study conclusion to be more valid than those in the 'with' group (mean difference: 0.97, 95% CI 0.59 to 1.36, P<0.001).

Conclusion: The overstatements in abstract conclusions did not significantly influence the primary care physicians' evaluations of the intervention effect when necessary information about the primary outcomes was distinctly reported.

Trial registration number: UMIN000025317; Pre-results.

Keywords: clinical trial; general practice; overstatements; primary care physicians; randomised controlled trials; reporting bias.

Figure 1

Flow diagram of participants. JPCA, Japan Primary Care Association.

Figure 2

Evaluation of the beneficial effect and validity of the intervention discussed in the abstract. The answers to q1 “How beneficial do you think the intervention A is to patients?”, and q2 “How valid is this conclusion in your opinion?” given in a scale of 0 (not at all) to 10 (very likely). Boxes showed the median score (horizontal rule) with 25th and 75th percentile.