Benchmarking of Machine Learning classifiers on plasma proteomic for COVID-19 severity prediction through interpretable artificial intelligence

Abstract

The SARS-CoV-2 pandemic highlighted the need for software tools that could facilitate patient triage regarding potential disease severity or even death. In this article, an ensemble of Machine Learning (ML) algorithms is evaluated in terms of predicting the severity of their condition using plasma proteomics and clinical data as input. An overview of AI-based technical developments to support COVID-19 patient management is presented outlining the landscape of relevant technical developments. Based on this review, the use of an ensemble of ML algorithms that analyze clinical and biological data (i.e., plasma proteomics) of COVID-19 patients is designed and deployed to evaluate the potential use of AI for early COVID-19 patient triage. The proposed pipeline is evaluated using three publicly available datasets for training and testing. Three ML "tasks" are defined, and several algorithms are tested through a hyperparameter tuning method to identify the highest-performance models. As overfitting is one of the typical pitfalls for such approaches (mainly due to the size of the training/validation datasets), a variety of evaluation metrics are used to mitigate this risk. In the evaluation procedure, recall scores ranged from 0.6 to 0.74 and F1-score from 0.62 to 0.75. The best performance is observed via Multi-Layer Perceptron (MLP) and Support Vector Machines (SVM) algorithms. Additionally, input data (proteomics and clinical data) were ranked based on corresponding Shapley additive explanation (SHAP) values and evaluated for their prognosticated capacity and immuno-biological credence. This "interpretable" approach revealed that our ML models could discern critical COVID-19 cases predominantly based on patient's age and plasma proteins on B cell dysfunction, hyper-activation of inflammatory pathways like Toll-like receptors, and hypo-activation of developmental and immune pathways like SCF/c-Kit signaling. Finally, the herein computational workflow is corroborated in an independent dataset and MLP superiority along with the implication of the abovementioned predictive biological pathways are corroborated. Regarding limitations of the presented ML pipeline, the datasets used in this study contain less than 1000 observations and a significant number of input features hence constituting a high-dimensional low-sample (HDLS) dataset which could be sensitive to overfitting. An advantage of the proposed pipeline is that it combines biological data (plasma proteomics) with clinical-phenotypic data. Thus, in principle, the presented approach could enable patient triage in a timely fashion if used on already trained models. However, larger datasets and further systematic validation are needed to confirm the potential clinical value of this approach. The code is available on Github: https://github.com/inab-certh/Predicting-COVID-19-severity-through-interpretable-AI-analysis-of-plasma-proteomics.

Keywords: Artificial intelligence; COVID-19; Forecasting; Machine Learning; Severity prediction.

Graphical abstract

Fig. 1

Methodology (part A: The overall approach of the methodology, part B: The Machine Learning tasks).

Fig. 2

Development and evaluation of the final models A Data preprocessing B Feature contribution on the models' results.

Fig. 3

Task 1 performance of F1-score from GridSearch hyperparameter tuning A. MGH B. YPS C. ICL.

Fig. 4

Task 2 performance of F1-score from GridSearch hyperparameter tuning for trained model in MGH dataset and test in ICL and YPS dataset A. YPS B. ICL.

Fig. 5

Task 2 performance of F1-score from GridSearch hyperparameter tuning for trained model in YPS dataset and test in ICL and MGH dataset A. ICL B. MGH.

Fig. 6

Task 2 performance of F1-score from GridSearch hyperparameter tuning for trained model in ICL dataset and test in YPS and MGH dataset A. YPS B. MGH.

Fig. 7

Task 3 performance of F1-score from GridSearch hyperparameter tuning.

Fig. 8

MLP predicts COVID-19 severity using proteins regulating immune-inflammatory and developmental pathways in Task 1. GeneMania protein-protein interaction networks (PPI), pathway enrichment dotplots and heatmaps with protein-nodes for the most predictive plasma proteins (top-30) as features of MLP (Task 1) considering the MGH (A), YPS (B) and ICL (C) datasets respectively.

Supplementary Fig. 2

Task 1 performance from GridSearch hyperparameter tuning A. MGH dataset B. YPS dataset C. ICL dataset.

Supplementary Fig. 3

Task 2 performance from GridSearch hyperparameter tuning A. The model is trained in MGH dataset and test in ICL dataset B. The model is trained in MGH dataset and test in YPS dataset C. The model is trained in YPS dataset and test in ICL dataset.

Supplementary Fig. 4

Task 2 performance from GridSearch hyperparameter tuning A. The model is trained in YPS dataset and test in MGH dataset B. The model is trained in ICL dataset and test in YPS dataset C. The model is trained in ICL dataset and test in MGH dataset.

Supplementary Fig. 5

Task 3 performance from GridSearch hyperparameter tuning.

Supplementary Fig. 6

SHAP values of ML models that are presented the contribution of PCA components to the final classification. A. SHAP values of Task 1 MLP model trained with ICL dataset B. SHAP values of Task 2 SVM model trained with ICL dataset and tested to MGH C. SHAP values of Task 2 SVM model trained with ICL dataset and tested to YPS D. SHAP values of Task 1 MLP model trained with MGH dataset E. SHAP values of Task 2 MLP model trained with MGH dataset and tested to ICL F. SHAP values of Task 2 MLP model trained with MGH dataset and tested to YPS.

Supplementary Fig. 7

SHAP values of ML models that are presented the contribution of PCA components to the final classification. A. SHAP values of Task 1 MLP model trained with YPS dataset B. SHAP values of Task 2 MLP model trained with YPS dataset and tested to ICL C. SHAP values of Task 2 MLP model trained with YPS dataset and tested to MGH D. SHAP values of Task 3 SVM model.

Supplementary Fig. 10

MLP predicts COVID-19 severity in the independent MC dataset, using similar proteins and pathways to MGH-YPS-ICL datasets. GeneMania protein-protein interaction networks (PPI), pathway enrichment dotplots and heatmaps with protein-nodes for the most predictive plasma proteins (top-30) as features of MLP (Task 1) considering for the validation dataset MC.

Supplementary Fig. 11

SHAP values of ML models that are presented the contribution of PCA components to the final classification. A. SHAP values of MLP model in evaluation dataset MC.

Supplementary Fig. 12

Evaluation task performance from GridSearch hyperparameter tuning.

Supplementary Fig. 13

The proposed pipeline of our study is evaluated through the MC dataset.