Diagnosis of neurosyphilis in HIV-negative patients with syphilis: development, validation, and clinical utility of a suite of machine learning models

Abstract

Background: The ability to accurately identify the absolute risk of neurosyphilis diagnosis for patients with syphilis would allow preventative and therapeutic interventions to be delivered to patients at high-risk, sparing patients at low-risk from unnecessary care. We aimed to develop, validate, and evaluate the clinical utility of simplified clinical diagnostic models for neurosyphilis diagnosis in HIV-negative patients with syphilis.

Methods: We searched PubMed, China National Knowledge Infrastructure and UpToDate for publications about neurosyphilis diagnostic guidelines in English or Chinese from database inception until March 15, 2023. We developed and validated machine learning models with a uniform set of predictors based on six authoritative diagnostic guidelines across four continents to predict neurosyphilis using routinely collected data from real-world clinical practice in China and the United States (through the Dermatology Hospital of Southern Medical University in Guangzhou [659 recruited between August 2012 and March 2022, treated as Development cohort], the Beijing Youan Hospital of Capital Medical University in Beijng [480 recruited between December 2013 and April 2021, treated as External cohort 1], the Zhongshan Hospital of Xiamen University in Xiamen [493 recruited between November 2005 and November 2021, treated as External cohort 2] from China, and University of Washington School of Medicine in Seattle [16 recruited between September 2002 and April 2014, treated as External cohort 3] from United States). We included all these patients with syphilis into our analysis, and no patients were further excluded. We trained eXtreme gradient boosting (XGBoost) models to predict the diagnostic outcome of neurosyphilis according to each diagnostic guideline in two scenarios, respectively. Model performance was measured through both internal and external validation in terms of discrimination and calibration, and clinical utility was evaluated using decision curve analysis.

Findings: The final simplified clinical diagnostic models included neurological symptoms, cerebrospinal fluid (CSF) protein, CSF white blood cell, and CSF venereal disease research laboratory test/rapid plasma reagin. The models showed good calibration with rescaled Brier score of 0.99 (95% CI 0.98-1.00) and excellent discrimination (the minimum value of area under the receiver operating characteristic curve, 0.84; 95% CI 0.81-0.88) when externally validated. Decision curve analysis demonstrated that the models were useful across a range of neurosyphilis probability thresholds between 0.33 and 0.66 compared to the alternatives of managing all patients with syphilis as if they do or do not have neurosyphilis.

Interpretation: The simplified clinical diagnostic models comprised of readily available data show good performance, are generalisable across clinical settings, and have clinical utility over a broad range of probability thresholds. The models with a uniform set of predictors can simplify the sophisticated clinical diagnosis of neurosyphilis, and guide decisions on delivery of neurosyphilis health-care, ultimately, support accurate diagnosis and necessary treatment.

Funding: The Natural Science Foundation of China General Program, Health Appropriate Technology Promotion Project of Guangdong Medical Research Foundation, Department of Science and technology of Guangdong Province Xinjiang Rural Science and Technology（Special Commissioner）Project, Southern Medical University Clinical Research Nursery Garden Project, Beijing Municipal Administration of Hospitals Incubating Program.

Keywords: Clinical diagnostic model; Diagnosis; Neurosyphilis; Risk-differentiated; Syphilis.

Fig. 1

Schematic overview of the framework of this study. Tenfold cross-validation was used to evaluate performance of each model, avoid any overfitting/underfitting, ensure robustness of models and minimize bias. Specifically, an inner tenfold cross-validation was applied to tune the hyperparameters with a random gird search, set to maximize the area under the receiver operating characteristic curve (AUROC). The two steps of tenfold cross-validation constituted the double tenfold cross-validation in our study to minimize bias in performance evaluation. We trained XGBoost models to predict the diagnostic outcome of neurosyphilis using the full available features in Development cohort (Table 1) according to each diagnostic guideline in two scenarios, respectively. To allow for interpretation of our models' predictions, we assessed feature importance, respectively, using the Shapley values to identify a feature's relative contribution to uncover key features. Based on the rankings of feature importance from the models in two scenarios, respectively, we selected a panel of consensus-based key features (i.e., a consensus reached by comprehensive considerations on the rankings of feature importance in all models developed from the diagnostic guidelines and ready availability and accessibility in real-world clinical practice) as a panel of key drivers for clinical diagnosis of neurosyphilis. We retrained our model using this subset of features (i.e., the panel of consensus-based key features), and arrived at simplified clinical diagnostic models through three external validation cohorts to identify patients with neurosyphilis. An online browser-accessible version of the final simplified clinical diagnostic models was also made available for external use (https://zhen-lu.shinyapps.io/Machine-learning-based-diagnosis-for-neurosyphilis/).

Fig. 2

The Shapley values for each of candidate features in Scenario 1 and 2. To allow for interpretation of our models' predictions, we assessed feature importance, respectively, using the Shapley values to identify a feature's relative contribution to uncover key features. Based on the rankings of feature importance from the models in two scenarios, respectively, we selected a panel of consensus-based key features (i.e., a consensus reached by comprehensive considerations on the rankings of feature importance in all models developed from the diagnostic guidelines and ready availability and accessibility in real-world clinical practice) as a panel of key drivers for clinical diagnosis of neurosyphilis. Each point in the plots is a Shapley (importance) value for a single patient. The color of each point represents the magnitude and direction of the value of that feature for that patient. A feature for a patient with a Shapley value below zero decreases the probability specifying the likelihood of being diagnosed as neurosyphilis, in addition, a higher probability indicates a higher likelihood of neurosyphilis. A in the figure indicates the results for the China 2020 model; B in the figure indicates the results for the Europe 2020 model; C in the figure indicates the results for the NT Australia 2022 model; D in the figure indicates the results for the UpToDate 2020 model; E in the figure indicates the results for the US CDC 2018 model; F in the figure indicates the results for the US 2021 model.

Fig. 3

Predictive performance of six simplified models in the internal and external validation datasets in Scenario 1. The lines with different colors in the plots represented the values of area under the receiver operating characteristic curve for each model. Several receiver operating characteristic curves overlapped completely due to the same values of area under the receiver operating characteristic curve. Predictive performance of six simplified models in the internal and external validation datasets in Scenario 2 were shown in Supplementary Figure S4. Abbreviations: AUROC, Area under the receiver operating characteristic curve; CI, Confidence interval; NA, Not applicable. A glossary of terms of statistics and machine learning used in this study could be found in the Supplementary Table S5. A in the figure indicates the results for the China 2020 model; B in the figure indicates the results for the Europe 2020 model; C in the figure indicates the results for the NT Australia 2022 model; D in the figure indicates the results for the UpToDate 2020 model; E in the figure indicates the results for the US CDC 2018 model; F in the figure indicates the results for the US 2021 model.

Fig. 4

Decision curve analysis using the simplified models in the internal validation dataset from Development cohort in Scenario 1 and 2 to guide decisions at health-care systems and individual shared decision-making levels. We assessed the potential clinical utility of the models (i.e., the net benefit of the models) by using a decision curve analysis. The net benefit of the models incorporating the trade-offs between true-positives and false-positives for a wide range of clinical probability thresholds is considered by the decision curve analysis. Thus, the decision curve analysis could consider the benefits and harms of using a model for clinical decision making, which allows decisions on management of patients with syphilis with variable neurosyphilis risk probabilities. A in the figure indicates the results for the China 2020 model; B in the figure indicates the results for the Europe 2020 model; C in the figure indicates the results for the NT Australia 2022 model; D in the figure indicates the results for the UpToDate 2020 model; E in the figure indicates the results for the US CDC 2018 model; F in the figure indicates the results for the US 2021 model.