A scoping review of artificial intelligence applications in clinical trial risk assessment

Abstract

Artificial intelligence (AI) is increasingly applied to clinical trial risk assessment, aiming to improve safety and efficiency. This scoping review analyzed 142 studies published between 2013 and 2024, focusing on safety (n = 55), efficacy (n = 46), and operational (n = 45) risk prediction. AI techniques, including traditional machine learning, deep learning (e.g., graph neural networks, transformers), and causal machine learning, are used for tasks like adverse drug event prediction, treatment effect estimation, and phase transition prediction. These methods utilize diverse data sources, from molecular structures and clinical trial protocols to patient data and scientific publications. Recently, large language models (LLMs) have seen a surge in applications, featuring in 7 out of 33 studies in 2023. While some models achieve high performance (AUROC up to 96%), challenges remain, including selection bias, limited prospective studies, and data quality issues. Despite these limitations, AI-based risk assessment holds substantial promise for transforming clinical trials, particularly through improved risk-based monitoring frameworks.

Fig. 1. PRISMA flowchart describing the source of evidence retrieval and selection process.

From the 4,328 manuscripts identified during the search phase, 3302 titles and abstracts were screened after de-duplication, and 194 full texts. A total of 142 studies were included for full-text analysis.

Fig. 2. AI applications for risk assessment in clinical trials.

AI applications can be categorized into safety, efficacy, and operational risk assessment. They follow a typical three-step analysis approach: representation, learning, and prediction (or inference). In the first step, clinical trial-related information, such as compound, participants, and protocol, is encoded as vectors. In the second step, models are learned to infer risks. In the last step, different risk types are predicted, and performance metrics are obtained.

Fig. 3. Trend of AI risk prediction for clinical trials.

Distribution of published studies a over time, and categorized by b subject area, c country, and d venue type. The drop in the number of studies in 2024 is due to the cut-off date - 15.07.2024 - in the study selection criteria.

Fig. 4. Distribution of studies according to phase, condition, and safety concerns.

a Number of studies focused on phases I-IV. b Number of phases per study. c Conditions in condition-specific studies. d Safety categories in safety-specific studies.

Fig. 5. Machine learning models used in risk assessment of clinical trials.

a Trends of the different types of machine learning approaches for risk prediction of clinical trials. b Approaches used for the different risk assessment tasks. c Algorithms with results published in at least 10 risk assessment studies. d Trends in utilizing large language models for clinical trial risk assessment. The drop in the number of studies in 2024 is due to the cut-off date - 15.07.2024 - in the study selection criteria.

Fig. 6. Datasets and metrics used to train and evaluate risk prediction models.

a Distribution of studies using (only) public and (at least one) private dataset. b Number of compounds, participants, and protocols used to train and evaluate models. c Top-10 metrics used in clinical trial risk assessment. d Number of reported metrics per study.

Fig. 7. Model performance and dataset size.

a Model performance across ADE (safety), outcome (efficacy), and phase success (operational) risk assessment using SIDER, individual trial data, and clinical trial protocols, respectively. b Number of instances used in the ADE, outcome, and phase success prediction tasks. Transparent Horizontal lines in (a) represent the median performance for each prediction task: ADE (red), outcome (grey), and phase success (blue).