A comparison of machine learning methods to find clinical trials for inclusion in new systematic reviews from their PROSPERO registrations prior to searching and screening

Abstract

Searching for trials is a key task in systematic reviews and a focus of automation. Previous approaches required knowing examples of relevant trials in advance, and most methods are focused on published trial articles. To complement existing tools, we compared methods for finding relevant trial registrations given a International Prospective Register of Systematic Reviews (PROSPERO) entry and where no relevant trials have been screened for inclusion in advance. We compared SciBERT-based (extension of Bidirectional Encoder Representations from Transformers) PICO extraction, MetaMap, and term-based representations using an imperfect dataset mined from 3632 PROSPERO entries connected to a subset of 65,662 trial registrations and 65,834 trial articles known to be included in systematic reviews. Performance was measured by the median rank and recall by rank of trials that were eventually included in the published systematic reviews. When ranking trial registrations relative to PROSPERO entries, 296 trial registrations needed to be screened to identify half of the relevant trials, and the best performing approach used a basic term-based representation. When ranking trial articles relative to PROSPERO entries, 162 trial articles needed to be screened to identify half of the relevant trials, and the best-performing approach used a term-based representation. The results show that MetaMap and term-based representations outperformed approaches that included PICO extraction for this use case. The results suggest that when starting with a PROSPERO entry and where no trials have been screened for inclusion, automated methods can reduce workload, but additional processes are still needed to efficiently identify trial registrations or trial articles that meet the inclusion criteria of a systematic review.

Keywords: clinical trials; information retrieval; systematic reviews.

Figure 1.

A schematic representation of the traditional systematic review of clinical trials compared to proactively allocating trials to systematic review questions before the trials are completed (left), and the types of tools used to support automation (right), including: (a) active learning approaches where experts label trials during model training; (b) proactive identification of trial registrations and trial articles to update systematic reviews where some included trials are already known; and (c) proactive identification of trial registrations and trial articles for new systematic review questions where experts are not available to identify example trials before or during model training.

Figure 2.

An example of the comparison framework using mined data for ranking trial registrations to PROSPERO entries, including the steps of processing (grey), numbers of trial registrations (green), PROSPERO entries (blue), and known connections (red). Other combinations of PROSPERO entries, systematic review articles, trial articles, and trial registrations are included in the Supplementary Material.

Figure 3.

The scatter plot of the TF-IDF methods using different targeted terms with cosine similarity as ranking score in the mined data for: (a) PROSPERO entry to trial registration; (b) PROSPERO entry to trial article; (c) systematic review article to trial registration; and (d) systematic review article to trial article.

Figure 4.

The Scatter plot of the TF-IDF methods using different targeted terms with cosine similarity as ranking score in manually curated data for: (a) PROSPERO entry to trial registration; (b) PROSPERO entry to trial article; (c) systematic review article to trial registration; and (d) systematic review article to trial article.