Wasted research when systematic reviews fail to provide a complete and up-to-date evidence synthesis: the example of lung cancer

Abstract

Background: Multiple treatments are frequently available for a given condition, and clinicians and patients need a comprehensive, up-to-date synthesis of evidence for all competing treatments. We aimed to quantify the waste of research related to the failure of systematic reviews to provide a complete and up-to-date evidence synthesis over time.

Methods: We performed a series of systematic overviews and networks of randomized trials assessing the gap between evidence covered by systematic reviews and available trials of second-line treatments for advanced non-small cell lung cancer. We searched the Cochrane Database of Systematic Reviews, Database of Abstracts of Reviews of Effects, MEDLINE, EMBASE, and other resources sequentially by year from 2009 to March 2, 2015. We sequentially compared the amount of evidence missing from systematic reviews to the randomized evidence available for inclusion each year. We constructed cumulative networks of randomized evidence over time and evaluated the proportion of trials, patients, treatments, and treatment comparisons not covered by systematic reviews on December 31 each year from 2009 to 2015.

Results: We identified 77 trials (28,636 patients) assessing 47 treatments with 54 comparisons and 29 systematic reviews (13 published after 2013). From 2009 to 2015, the evidence covered by existing systematic reviews was consistently incomplete: 45 % to 70 % of trials; 30 % to 58 % of patients; 40 % to 66 % of treatments; and 38 % to 71 % of comparisons were missing. In the cumulative networks of randomized evidence, 10 % to 17 % of treatment comparisons were partially covered by systematic reviews and 55 % to 85 % were partially or not covered.

Conclusions: We illustrate how systematic reviews of a given condition provide a fragmented, out-of-date panorama of the evidence for all treatments. This waste of research might be reduced by the development of live cumulative network meta-analyses.

Fig. 1

Flow diagram of selection of systematic reviews and randomized controlled trials of second-line treatments in advanced non-small cell lung cancer. ^*Additional full-text articles not identified by searching bibliographical databases; ^$63 full-text articles and 44 conference abstracts; ^¤69 full-text articles, 70 conference abstracts, 28 posted results, and 5 industry/FDA reports

Fig. 2

Network of 77 randomized controlled trials of second-line treatments in advanced non-small cell lung cancer. The thickness of connecting lines indicates the number of available comparisons. The size of each node is proportional to the number of patients allocated to the corresponding treatment. AFL: aflibercept; AMR: amrubicin; ARQ197: tivantinib; BEV: bevacizumab; BIBF1120: nintedanib; BSC: best supportive care; CET: cetuximab; DAC: dacomitinib; DOC: docetaxel; ERL: erlotinib; EVE: everolimus; FIGI: figitumumab; FULV: fulvestrant; GEF: gefitinib; ICO: icotinib; MAT: matuzumab; MK-0646: dalotuzumab; ONA: onartuzumab; PAZ: pazopanib; PBO: placebo; PDX: pralatrexate; PEM: pemetrexed; PTX: paclitaxel; RAM: ramucirumab; SEL: selumetinib; SOR: sorafenib; SUN: sunitinib; TOP: topotecan; TRA: trametinib; VAN: vandetanib; VFL: vinflunine; VIN: vinorelbine

Fig. 3

Amount of treatments, treatment comparisons, trials, and patients not covered by systematic reviews from 2009 to 2015. ^*The last search for randomized trials and systematic reviews was conducted on March 2, 2015

Fig. 4

Cumulative networks of evidence showing the gap between the amount of randomized evidence covered by systematic reviews and the amount of randomized evidence available for inclusion. (a) 2009–2012 and (b) 2013–2015. ^*The last search for randomized trials and systematic reviews was conducted on March 2, 2015. From 2009 to 2015, we compared randomized controlled trials selected by systematic reviews published up to December 31 each year (up to March 2 for 2015) to all trials eligible for inclusion (i.e., all trial results published up to July 1 each year [up to August 31, 2014 for 2015]). Each node size is proportional to the total number of patients randomly allocated to the corresponding treatment across all randomized trials available for inclusion; we represented the proportion of randomized patients actually covered by systematic reviews by pie charts overlaid on nodes in the network. The thickness of each edge is proportional to the total number of randomized controlled trials between the corresponding treatments available for inclusion; we represented the proportion of trials actually selected by systematic reviews by a percentage bar chart overlaid on edges in the network

Fig. 5

A new approach to synthesize evidence: live cumulative network meta-analysis. Starting from an initial NMA, a research community would regularly (e.g., every 3 months), search for, screen, and select trials with new results and, if any, extract data, assess the risk of bias, and update the NMA. NMA: network meta-analysis