Current Status and Future Opportunities in Modeling Clinical Characteristics of Multiple Sclerosis

Abstract

Development of effective treatments requires understanding of disease mechanisms. For diseases of the central nervous system (CNS), such as multiple sclerosis (MS), human pathology studies and animal models tend to identify candidate disease mechanisms. However, these studies cannot easily link the identified processes to clinical outcomes, such as MS severity, required for causality assessment of candidate mechanisms. Technological advances now allow the generation of thousands of biomarkers in living human subjects, derived from genes, transcripts, medical images, and proteins or metabolites in biological fluids. These biomarkers can be assembled into computational models of clinical value, provided such models are generalizable. Reproducibility of models increases with the technical rigor of the study design, such as blinding, control implementation, the use of large cohorts that encompass the entire spectrum of disease phenotypes and, most importantly, model validation in independent cohort(s). To facilitate the growth of this important research area, we performed a meta-analysis of publications (n = 302) that model MS clinical outcomes extracting effect sizes, while also scoring the technical quality of the study design using predefined criteria. Finally, we generated a Shiny-App-based website that allows dynamic exploration of the data by selective filtering. On average, the published studies fulfilled only one of the seven criteria of study design rigor. Only 15.2% of the studies used any validation strategy, and only 8% used the gold standard of independent cohort validation. Many studies also used small cohorts, e.g., for magnetic resonance imaging (MRI) and blood biomarker predictors, the median sample size was <100 subjects. We observed inverse relationships between reported effect sizes and the number of study design criteria fulfilled, expanding analogous reports from non-MS fields, that studies that fail to limit bias overestimate effect sizes. In conclusion, the presented meta-analysis represents a useful tool for researchers, reviewers, and funders to improve the design of future modeling studies in MS and to easily compare new studies with the published literature. We expect that this will accelerate research in this important area, leading to the development of robust models with proven clinical value.

Keywords: MS disability; MS severity; clinical outcomes; machine learning; multiple sclerosis (MS); predictive models; reproducibility; technical quality.

Figure 1

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) chart summarizing the disposition of records identified from PubMed searches. The searches identified 782 records, of which 663 were unique. After several exclusion criteria defined in the figure, 302 unique records were included in the review.

Figure 2

Distribution of modeled clinical outcomes. The large pie chart in the center of the figure shows the distribution of categories of modeled outcomes. The three most frequent outcome categories are disability (37%, red), severity (26%, purple), and patient-reported outcomes (PROs; 15%, teal). The surrounding bar plots show the breakdown of each of these three categories.

Figure 3

Important characteristics of the studies reviewed. (A) Number of studies (x-axis) per predictor type (y-axis). (B) Number of subjects (y-axis) by predictor type (x-axis). (C) Percentage of studies (x-axis; number of studies in parentheses) fulfilling each preselected criteria of experimental design/technical quality of the study (y-axis). (D) Percentage of studies (y-axis) per each predictor type fulfilling a number of technical criteria (x-axis).

Figure 4

Relationship between technical quality of the study and reported effect size. The proportion of studies fulfilling the sum of the seven technical quality criteria (zero weakest experimental design to seven strongest experimental design) for 253 studies with training cohorts (A) and cross-validation cohorts (B). The number of studies in each category is listed above the bars. Effect sizes reported by studies categorized based on the number of technical quality criteria they fulfilled (0–7) for training cohort (C; n = 253) and cross-validation (D; n = 28) results. In both cohorts, the reported effect sizes decreased as the number of technical quality criteria fulfilled by these studies increased.

Figure 5

Relationship between technical quality and sample size of the study and reported effect sizes. (A) Study quality is defined by the number of subjects and number of criteria fulfilled with high quality studies falling 1 SD above both criteria and low-quality studies falling 1 SD below both criteria. (B) Boxplot compares the normalized effect sizes between low- and high-quality studies using a two-sample Wilcoxon (Mann–Whitney) test. Low-quality studies were found to have higher effect sizes at a significant p-value of 0.017.