CAPI-Detect: machine learning in capillaroscopy reveals new variables influencing diagnosis

Abstract

Objectives: Nailfold videocapillaroscopy (NVC) is the gold standard for diagnosing SSc and differentiating primary from secondary RP. The CAPI-Score algorithm, designed for simplicity, classifies capillaroscopy scleroderma patterns (CSPs) using a limited number of capillary variables. This study aims to develop a more advanced machine learning (ML) model to improve CSP identification by integrating a broader range of statistical variables while minimizing examiner-related bias.

Methods: A total of 1780 capillaroscopies were randomly and blindly analysed by three to four trained observers. Consensus was defined as agreement among all but one observer (partial consensus) or unanimous agreement (full consensus). Capillaroscopies with at least partial consensus were used to train ML-based classification models using CatBoost software, incorporating 24 capillary architecture-related variables extracted via automated NVC analysis. Validation sets were employed to assess model performance.

Results: Of the 1490 capillaroscopies classified with consensus, 515 achieved full consensus. The model, evaluated on partial and full consensus datasets, achieved 0.912, 0.812 and 0.746 accuracy for distinguishing SSc from non-SSc, among SSc patterns, and between normal and non-specific patterns, respectively. When evaluated on full consensus only, accuracy improved to 0.910, 0.925 and 0.933. CAPI-Detect outperformed CAPI-Score, revealing novel capillary variables critical to ML-based classification.

Conclusions: CAPI-Detect, an ML-based model, provides an unbiased, quantitative analysis of capillary structure, shape, size and density, significantly improving capillaroscopic pattern identification.

Keywords: CatBoost algorithm; disease pattern; examiner consensus; machine learning-based model; nailfold capillaroscopy; quantitative analysis; software-based analysis; systemic sclerosis.

Figure 1.

Flowchart diagram of the study. Capillaroscopies with partial and full examiner consensus (considered the gold standard) to be software-analysed and, subsequently, used to train the machine learning–based models. Approximately 20% of capillaroscopies were reserved as a validation set, not used for training. For additional evaluation, the validation set was restricted to cases with full examiner consensus only, allowing for an assessment of matches and discrepancies between manual analysis and the machine learning–based algorithm. This restriction ensured a focus on higher-quality cases with clearer classification among examiners. The percentage of each disease pattern in the training and validation sets is indicated in parentheses. Nor: normal; NS: non-specific; SSc-a: active scleroderma pattern; SSc-e: early scleroderma pattern; SSc-l: late scleroderma pattern

Figure 2.

Weight of the software-assessed variables on the machine learning–based models. The weights of each variable for the three models that form the algorithm are shown for the steps to discriminate between non-SSc and SSc pattern (A), between early, active and late within SSc patterns (B), and between normal and non-specific within non-SSc patterns (C). Variables are arranged in decreasing order of weight. MAD: mean apical diameter; MLW: mean limb width

Figure 3.

Precision-recall curves for the trained models. Precision-recall curves evaluated the accuracy of trained models using the validation set, and the validation set limited to full consensus cases, which consisted of 298 and 100 randomly selected capillaroscopies, respectively. AUPRC values corresponding to each disease pattern are shown. AUPRC: area under the precision-recall curve

Figure 4.

Confusion matrix comparing CAPI-Detect and CAPI-Score algorithm performance on the validation set for five-way pattern classification task. Two validation sets were assembled by randomly selecting 298 and 100 capillaroscopies with partial and full consensus, and full consensus only among examiners, respectively. Performance of machine learning-based CAPI-Detect and hand-crafted CAPI-Score algorithms for the five-way classification task [normal, non-specific (NS), early, active and late patterns] was evaluated with the validation sets. Confusion matrices display the performance of both algorithms. Accuracy for each pattern is indicated in percentages on the right-hand side of each matrix row