The interpretation of systematic reviews with meta-analyses: an objective or subjective process?

Abstract

Background: Discrepancies between the conclusions of different meta-analyses (quantitative syntheses of systematic reviews) are often ascribed to methodological differences. The objective of this study was to determine the discordance in interpretations when meta-analysts are presented with identical data.

Methods: We searched the literature for all randomized clinical trials (RCT) and review articles on the efficacy of intravenous magnesium in the early post-myocardial infarction period. We organized the articles chronologically and grouped them in packages. The first package included the first RCT, and a summary of the review articles published prior to first RCT. The second package contained the second and third RCT, a meta-analysis based on the data, and a summary of all review articles published prior to the third RCT. Similar packages were created for the 5th RCT, 10th RCT, 20th RCT and 23rd RCT (all articles). We presented the packages one at a time to eight different reviewers and asked them to answer three clinical questions after each package based solely on the information provided. The clinical questions included whether 1) they believed magnesium is now proven beneficial, 2) they believed magnesium will eventually be proven to be beneficial, and 3) they would recommend its use at this time.

Results: There was considerable disagreement among the reviewers for each package, and for each question. The discrepancies increased when the heterogeneity of the data increased. In addition, some reviewers became more sceptical of the effectiveness of magnesium over time, and some reviewers became less sceptical.

Conclusion: The interpretation of the results of systematic reviews with meta-analyses includes a subjective component that can lead to discordant conclusions that are independent of the methodology used to obtain or analyse the data.

Figure 1

Forest plots that illustrate the data are shown for each of the meta-analysis packages, along with the fixed and random effects odds ratios at each stage. The numbers in the graph are slightly different than those found in Table 1 because some data entry errors were discovered at some time points and corrected only after some members reviewed the package. The difference in the numbers between Table 1 and Figure 1 are minor and would not be expected to alter the responses by our reviewers. To remain consistent and avoid confusion, we have provided only the corrected data in Figure 1, and the numbers provided to the reviewers in Table 1. The first error was in Rasmussen 1986 article where the proportions were entered as the raw numbers instead of the frequencies. This was corrected by the next package. A 1988 paper by Rasmussen that was a follow-up analysis based on some criticisms in letters to the editors was included in packages RCT 5 and RCT 10 but then omitted later on. Finally, the numbers for the Thorgersen 1993 paper were correct in the RCT 10 package but were incorrect in the RCT 20 package due to a transcription error when we switched software; this was corrected for the RCT23 package.