The automation of relevant trial registration screening for systematic review updates: an evaluation study on a large dataset of ClinicalTrials.gov registrations

Abstract

Background: Clinical trial registries can be used as sources of clinical evidence for systematic review synthesis and updating. Our aim was to evaluate methods for identifying clinical trial registrations that should be screened for inclusion in updates of published systematic reviews.

Methods: A set of 4644 clinical trial registrations (ClinicalTrials.gov) included in 1089 systematic reviews (PubMed) were used to evaluate two methods (document similarity and hierarchical clustering) and representations (L2-normalised TF-IDF, Latent Dirichlet Allocation, and Doc2Vec) for ranking 163,501 completed clinical trials by relevance. Clinical trial registrations were ranked for each systematic review using seeding clinical trials, simulating how new relevant clinical trials could be automatically identified for an update. Performance was measured by the number of clinical trials that need to be screened to identify all relevant clinical trials.

Results: Using the document similarity method with TF-IDF feature representation and Euclidean distance metric, all relevant clinical trials for half of the systematic reviews were identified after screening 99 trials (IQR 19 to 491). The best-performing hierarchical clustering was using Ward agglomerative clustering (with TF-IDF representation and Euclidean distance) and needed to screen 501 clinical trials (IQR 43 to 4363) to achieve the same result.

Conclusion: An evaluation using a large set of mined links between published systematic reviews and clinical trial registrations showed that document similarity outperformed hierarchical clustering for identifying relevant clinical trials to include in systematic review updates.

Keywords: Document similarity; Hierarchical clustering; Systematic reviews; Trial registrations.

Fig. 1

Construction of the dataset using PubMed, CrossRef, and ClinicalTrials.gov.

Fig. 2

Illustration of the hierarchical agglomerative clustering method (left) and traversal on the resulting dendrogram (right)

Fig. 3

The median recall for 1089 systematic reviews after screening a given number of trial registrations

Fig. 4

Effect of seeding set size to the number of trials screened to achieve 95% recall

Fig. 5

The t-SNE visualization of the evaluated 4644 trials (blue) and the other ClinicalTrials.gov trials (grey)