vigiGrade: a tool to identify well-documented individual case reports and highlight systematic data quality issues

Abstract

Background: Individual case safety reports of suspected harm from medicines are fundamental to post-marketing surveillance. Their value is directly proportional to the amount of clinically relevant information they include. To improve the quality of the data, communication between stakeholders is essential and can be facilitated by a simple score and visualisation of the results.

Objective: The objective of this study was to propose a measure of completeness and identify predictors of well-documented reports, globally.

Methods: The Uppsala Monitoring Centre has developed the vigiGrade completeness score to measure the amount of clinically relevant information in structured format, without reflecting whether the information establishes causality between the drug and adverse event. The vigiGrade completeness score (C) starts at 1 for reports with information on time-to-onset, age, sex, indication, outcome, report type, dose, country, primary reporter and comments. For each missing dimension, a penalty is detracted which varies with clinical relevance. We classified reports with C > 0.8 as well-documented and identified all such reports in the WHO global individual case safety report database, VigiBase, from 2007 to January 2012. We utilised odds ratios with statistical shrinkage to identify subgroups with unexpectedly high proportions of well-documented reports.

Results: Altogether, 430,000 (13 %) of the studied reports achieved C > 0.8 in VigiBase. For VigiBase as a whole, the median completeness was 0.41 with an interquartile range of 0.26-0.63. Two out of three well-documented reports come from Europe, and two out of three from physicians. Among the countries with more than 1,000 reports in total, the highest rate of well-documented reports is 65 % in Italy. Tunisia, Spain, Portugal, Croatia and Denmark each have rates above 50 %, and another 20 countries have rates above 30 %. On the whole, 24 % of the reports from physicians are well-documented compared with only 4 % for consumers/non-health professionals. Notably, Denmark and Norway have more than 50 % well-documented reports from consumers/non-health professionals and higher rates than for physicians. The rate of well-documented reports for the E2B format is 11 % compared with 22 % for the older INTDIS (International Drug Information System) format. However, for E2B reports entered via the WHO programme's e-reporting system VigiFlow, the rate is 29 %.

Conclusion: Overall, only one report in eight provides the desired level of information, but much higher proportions are observed for individual countries. Physicians and e-reporting tools also generate greater proportions of well-documented reports overall. Reports from consumers/non-health professionals in specific regions have excellent quality, which illustrates their potential for the future. vigiGrade has already provided valuable information by highlighting data quality issues both in Italy and the USA.

Fig. 1

Four examples of penalties (or lack thereof) when there is imprecise or missing information on time-to-onset. ADR adverse drug reaction, TTO time-to-onset

Fig. 2

An example of how the vigiGrade completeness score is calculated for a report

Fig. 3

Distribution of completeness and the proportion of well-documented reports over time in VigiBase

Fig. 4

Empirical distribution of vigiGrade completeness across the 3.3 million reports in VigiBase between January 2007 and January 2012

Fig. 5

Countries with at least 1,000 reports in VigiBase between 2007 and 2012 ordered by the number of well-documented reports. Red text indicates the 30 countries that had significantly higher than expected numbers of well-documented reports (shrunk log odds ratio exceeding 0.5)

Fig. 6

Number of well-documented reports for different primary reporters

Fig. 7

The proportion of well-documented reports by country and primary reporter is marked by a black circle, the size corresponding to the number of reports. The grey bars represent the proportion of well-documented reports for the country overall and the dotted vertical line the proportion for the primary reporter overall

Fig. 8

The average completeness over time for reports from the USA. A noteworthy decline in 2011 was due to miscoding of the age unit format in their E2B reports. Historical data, not representative of US reports from this time period as represented in VigiBase today

Fig. 9

Outcome information as reported by Italy between 2007 and 2011. Historical data, not representative of Italian reports from this time period as represented in VigiBase today