Mapping Strategies to Assess and Increase the Validity of Published Disproportionality Signals: A Meta-Research Study

Abstract

Background and aim: Disproportionality analysis is traditionally used in spontaneous reporting systems to generate working hypotheses about potential adverse drug reactions: the so-called disproportionality signals. We aim to map the methods used by researchers to assess and increase the validity of their published disproportionality signals.

Methods: From a systematic literature search of published disproportionality analyses up until 1 January 2020, we randomly selected and analyzed 100 studies. We considered five domains: (1) rationale for the study, (2) design of disproportionality analyses, (3) case-by-case assessment, (4) use of complementary data sources, and (5) contextualization of the results within existing evidence.

Results: Among the articles, multiple strategies were adopted to assess and enhance the results validity. The rationale, in 95 articles, was explicitly referred to the accrued evidence, mostly observational data (n = 46) and regulatory documents (n = 45). A statistical adjustment was performed in 34 studies, and specific strategies to correct for biases were implemented in 33 studies. A case-by-case assessment was complementarily performed in 35 studies, most often by investigating temporal plausibility (n = 26). Complementary data sources were used in 25 articles. In 78 articles, results were contextualized using accrued evidence from the literature and regulatory documents, the most important sources being observational (n = 45), other disproportionalities (n = 37), and case reports (n = 36).

Conclusions: This meta-research study highlighted the heterogeneity in methods and strategies used by researchers to assess the validity of disproportionality signals. Mapping these strategies is a first step towards testing their utility in different scenarios and developing guidelines for designing future disproportionality analysis.

Fig. 1

Use of strategies to assess/enhance validity in the 100 disproportionality studies randomly selected, grouped by domain. The five colored bars represent the five domains. Each domain has a gauge plot on the left, showing the percentage of studies using at least a technique of the domain, and an UpSet plot on the right, showing the frequency of use of each technique (side bar plot) and of each combination (bar plot on the top). In some cases, a rationale or a literature support was provided based on preexisting works, for which the underlying evidence was not clearly specified, as it came from narrative reviews, opinions, and commentaries. Articles with underspecified rationale or literature support were counted in the gauge plot, but not reported in the UpSet plot. Legend: MT correction, correction for multiple testing; multi DA, multiple disproportionality analysis; multi SRS, multiple spontaneous reporting system. Created with biorender.com