Comparison of different Lunit INSIGHT CXR software versions when reading chest radiographs for tuberculosis

Abstract

New versions of computer-aided detection (CAD) software for chest X-ray (CXR) interpretation during tuberculosis (TB) screening are regularly released which purport to have incremental performance gains. No studies have independently assessed differences in software performance between the World Health Organization recommended INSIGHT CXR software (Lunit, South Korea). A well-characterized Digital Imaging and Communications in Medicine (DICOM) test library was compiled using data from a community-based TB screening initiative in Ho Chi Minh City, Viet Nam. The performance of Lunit CAD software versions 3.1.0.0 and 3.9.0.1 (newer version) were compared by measuring the area under the receiver operating characteristic curve (AUC), stratified by key clinical and demographic variables and using Xpert MTB/RIF Ultra (Ultra) test results as the reference standard. Median abnormality scores were compared using the Wilcoxon signed-rank test and performance characteristics were compared at clinically-relevant cut-off thresholds (e.g., 90% sensitivity) between the versions. The DICOM test library contained 2,708 participants, of whom 10.3% had a Mycobacterium tuberculosis (MTB) positive Ultra test result. The newer software version had a significantly higher AUC than its predecessor (AUC 0.76 vs 0.78, p = 0.029), and performed significantly better among people with a past history of TB (AUC 0.67 vs 0.73, p = 0.003), older individuals (0.75 vs 0.77, p = 0.040) and males (0.73 vs 0.76, p = 0.008). When using an cut-off threshold optimized for the older software version, the newer software was significantly less accurate than its predecessors. However, when the cut-off threshold was re-calibrated, there were no significant differences in sensitivity and specificity between the software versions. Although INSIGHT CXR v3.9.0.1 has some significantly improved performance characteristics compared to its predecessor, further studies should assess how these performance differences translate into real-world improvements during TB screening. As new CAD software versions are rolled out, cut-off thresholds must be re-calibrated to ensure the continued accuracy of CXR interpretation.

Fig 1. ROC curves for the two Lunit CAD software versions for A) all individuals and stratified by B) past history of TB, C) age group and D) sex.

Fig 2. Flow diagram of TB screening and constitution of the DICOM test library.