External validation of a multiple sclerosis treatment decision score using data from the ProVal-MS cohort study

Abstract

Background: The course of relapsing-remitting multiple sclerosis (RRMS), frequently preceded by the clinically isolated syndrome (CIS), is variable and challenging to predict. Given many treatment options available, prognostic algorithms are gaining importance in informing initial treatment decisions. However, to date, only a few externally validated exists. External validation, which involves the application of a model to independent data, is essential. Privacy-preserving federated analyses of individual-level data facilitate external validation using clinical datasets that are typically difficult to access.

Objectives: Using data from the ProVal-MS study to externally validate the multiple sclerosis treatment decision score (MS-TDS), a predictive algorithm for early RRMS and CIS. The MS-TDS predicts the probability of the occurrence of at least one new or enlarging T2 lesion within 6-24 months following the onset of the disease and supports choosing between initiating platform treatment or a 'wait-and-see' approach. A secondary objective is to demonstrate the feasibility of privacy-preserving federated concepts within the Data Integration for Future Medicine (DIFUTURE) consortium.

Design: Prospective, multicentric, non-interventional cohort study (ProVal-MS) within DIFUTURE.

Methods: The calibrated MS-TDS was evaluated using the area under the receiver operating characteristic curve (AUROC) and the Brier score in both pooled and distributed settings. A decision curve analysis (DCA) was used to evaluate the net benefit of treatment decisions made by the MS-TDS in comparison to those made by treating neurologists.

Results: Of the 271 individuals diagnosed with CIS or early RRMS, 202 (78.2%) received platform treatment, while 59 (21.8%) did not receive treatment. The AUROC was 0.561 (95% CI: 0.492-0.630) in the pooled analysis and 0.567 (95% CI: 0.496-0.634) in the distributed analysis. DCA demonstrated a net benefit that was commensurate with that achieved by decisions made by experienced neurologists.

Conclusion: The external validation of the MS-TDS demonstrated low, non-significant predictive performance; however, it may serve as a useful complement, particularly for less-experienced neurologists. The distributed validation was found to be both feasible and compliant with data protection regulations.

Keywords: DatatSHIELD; decision curve analysis; distributed/federated analysis; external validation; multiple sclerosis; patient preference; personalised medicine; privacy-preserving analysis; prognostic factors.

Figure 1.

Flowchart.

Figure 2.

Re-calibrated calibration curve of the MS-TDS. The grey line shows a locally estimated scatter plot smoothing line (LOESS) with 95% confidence levels (shaded grey area). Red line indicates optimal calibration. The predicted probabilities of at least one new or enlarging T2L between months 6 and 24 after baseline were between around 25% and 60%. The re-calibrated MS-TDS is well calibrated for T2L probabilities below 55%. Predicted and observed outcomes show good agreement. MS-TDS, multiple sclerosis treatment decision score; T2L, T2 lesion.

Figure 3.

Area under the receiver operating characteristic curve (AUROC). The multiple sclerosis treatment decision score does not discriminate well between persons with relapsing-remitting multiple sclerosis with and without new or newly enlarged T2 lesions. AUC, area under the curve.

Figure 4.

The figure illustrates a DCA following the approach by Vickers and colleagues. DCAs estimates the clinical value of prediction models by quantifying the net benefit across a range of threshold probabilities. Net benefit combines true positives and false positives into a single metric, allowing comparison between strategies while accounting for the relative harm of unnecessary treatment versus disease progression. The threshold probability represents the individual risk level at which treatment would be considered appropriate, based on the patient’s preferences regarding potential benefits and harms, in consultation with the physician’s clinical judgment. DCAs applies this personalised threshold to evaluate which treatment decision strategy yields the highest net benefit for a given patient. Here, the net benefit of initiating platform treatment under four different strategies are compared: treating all patients (red graph), treating none (green graph), treatment decisions made by physicians in the ProVal study (blue graph), and a hypothetical treatment decision based on the MS-TDS (purple graph). The x-axis represents the threshold probability (at least one new/newly enlarged T2 lesion – T2L – between months 6 and 24 after baseline): for each value along the axis, the strategy with the highest net benefit is considered optimal. The blue interval indicates the range of T2L probabilities predicted by the MS-TDS in the ProVal population. MS-TDS shows marginally better net-benefit compared to treating all, treating non, and physicians’ decisions. DCA, decision curve analysis; MS-TDS, multiple sclerosis treatment decision score; T2L, T2 lesion.