A multi-omics systems vaccinology resource to develop and test computational models of immunity

Abstract

Systems vaccinology studies have identified factors affecting individual vaccine responses, but comparing these findings is challenging due to varying study designs. To address this lack of reproducibility, we established a community resource for comparing Bordetella pertussis booster responses and to host annual contests for predicting patients' vaccination outcomes. We report here on our experiences with the "dry-run" prediction contest. We found that, among 20+ models adopted from the literature, the most successful model predicting vaccination outcome was based on age alone. This confirms our concerns about the reproducibility of conclusions between different vaccinology studies. Further, we found that, for newly trained models, handling of baseline information on the target variables was crucial. Overall, multiple co-inertia analysis gave the best results of the tested modeling approaches. Our goal is to engage community in these prediction challenges by making data and models available and opening a public contest in August 2024.

Keywords: Bordetella Pertussis; CP: immunology; CP: systems biology; Contest; Multi-omics integration; Prediction models; Reproducibility; Systems vaccinology; Vaccine response.

Graphical abstract

Figure 1

Outline for establishing the CMI-PB resource (A) Recruitment of human subjects and longitudinal specimen collection. (B) Generation of multi-omics data to obtain a comprehensive understanding of the collected specimens. (C) Implementation of a data standardization approach to ensure consistency and comparability of the generated data. (D) The resulting dataset is provided in training and test formats to enable contestants to develop their predictive models. (E) The CMI-PB resource website serves as a platform for hosting an annual prediction challenge, offering data visualization tools for generated data, and providing access to teaching materials and datasets.

Figure 2

Data processing, computable matrices, and prediction model generation (A) Generation of a harmonized dataset involved identifying shared features between the training and test datasets and filtering out low-information features. Literature-based models (team 1) used raw data from the database and applied data-formatting methods specified by existing models. JIVE and MCIA approaches (teams 2 and 3) utilized harmonized datasets for constructing their models. (B) Flowchart illustrates the steps involved in identifying baseline prediction models from the literature, creating a derived model based on the original models' specifications, and performing predictions as described by the authors. (C) The JIVE approach involved creating a subset of the harmonized dataset by including only subjects with data for all four assays. The JIVE algorithm was then applied to calculate 10 factors, which were subsequently used for making predictions. JIVE employed five different regression models for prediction purposes. (D) MCIA approach applied MICE imputation on the harmonized dataset and used these data for model construction. MCIA method was applied to the training dataset to construct 10 factors. Then, these 10 factors and feature scores from the test dataset were utilized to construct global scores for the test dataset. Lasso regression was applied to make predictions. MCIAplus model was constructed by including additional features (demographic, clinical features, and 14 task values) as factor scores, and it also utilized lasso regression to make predictions. The MCIA approach utilized MICE imputation on the harmonized dataset for model construction. The MCIA method employed the imputed training dataset to construct 10 factors. These 10 factors, along with feature scores from the test dataset, were used to construct global scores for the test dataset. Lasso regression was applied to make predictions. Additionally, the MCIAplus model incorporated additional features such as demographic, clinical features, and 14 task values as factor scores. Finally, lasso regression was employed for making predictions.

Figure 3

Evaluation of the prediction models submitted for the first CMI-PB challenge Model evaluation was performed using Spearman’s rank correlation coefficient between predicted ranks by a contestant and actual rank for each of (A) antibody titers, (B) immune cell frequencies, and (C) transcriptomics tasks. The number denotes Spearman rank correlation coefficient, while crosses represent any correlations that are not significant using p ≥ 0.05. The baseline and MCIAplus models outperformed other models for most tasks.