The Devil Is in the Details: Incomplete Reporting in Preclinical Animal Research

Abstract

Incomplete reporting of study methods and results has become a focal point for failures in the reproducibility and translation of findings from preclinical research. Here we demonstrate that incomplete reporting of preclinical research is not limited to a few elements of research design, but rather is a broader problem that extends to the reporting of the methods and results. We evaluated 47 preclinical research studies from a systematic review of acute lung injury that use mesenchymal stem cells (MSCs) as a treatment. We operationalized the ARRIVE (Animal Research: Reporting of In Vivo Experiments) reporting guidelines for pre-clinical studies into 109 discrete reporting sub-items and extracted 5,123 data elements. Overall, studies reported less than half (47%) of all sub-items (median 51 items; range 37-64). Across all studies, the Methods Section reported less than half (45%) and the Results Section reported less than a third (29%). There was no association between journal impact factor and completeness of reporting, which suggests that incomplete reporting of preclinical research occurs across all journals regardless of their perceived prestige. Incomplete reporting of methods and results will impede attempts to replicate research findings and maximize the value of preclinical studies.

Fig 1. The ARRIVE guidelines have six Sections: Title, Abstract, Introduction, Methods, Results, Discussion.

Each Section has at least one item (e.g. ethics statement) with a description which we operationalized into discrete yes/no sub-items.

Fig 2. The six ARRIVE Sections are listed around the circumference of the chart starting with Title at twelve o’clock.

The line represents the percentage of sub-items (e.g. species) reported for all studies per Section (e.g. Methods). For example, for the Section Title (84%) we summed the total number of reported ‘yes’ sub-items (119) and then divided it by the number of independent sub-items (3) multiplied by the total number of studies (47): 119/(3*47) = 0.84.

Fig 3. The 17 ARRIVE items are listed around the circumference of the chart starting with title at twelve o’clock.

The line represents the percentage of sub-items (e.g. species) reported for all studies per item (e.g. experimental animals). For example, for the item title (84%) we summed the total number of reported ‘yes’ sub-items (119) and then divided it by the number of independent sub-items (3) multiplied by the total number of studies (47): 119/(3*47) = 0.84.

Fig 4. The six NIH ‘core’ reporting items are listed around the circumference of the chart starting with replicates at twelve o’clock.

The line represents the percentage of ARRIVE sub-items (e.g. was a sample size calculation conducted) reported ‘yes’ for all studies that matched with each NIH core item. ARRIVE sub-items matched with NIH items are listed in S4 Table.

Fig 5. The ARRIVE sub-items that aligned with the NIH’s biological materials: animals reporting recommendation are listed around the circumference of the chart starting with species at twelve o’clock.

The line represents the percentage of 47 studies that reported the sub-item (e.g. all 47 studies reported the sub-item species).

Fig 6. The ARRIVE sub-items that aligned with the NIH’s biological materials: cell lines reporting recommendation are listed around the circumference of the chart starting with species source at twelve o’clock.

The line represents the percentage of 47 studies that reported the sub-item (e.g. 96% of studies reported the sub-item species source).