Improving the Efficacy of ACR TI-RADS Through Deep Learning-Based Descriptor Augmentation

Abstract

Thyroid nodules occur in up to 68% of people, 95% of which are benign. Of the 5% of malignant nodules, many would not result in symptoms or death, yet 600,000 FNAs are still performed annually, with a PPV of 5-7% (up to 30%). Artificial intelligence (AI) systems have the capacity to improve diagnostic accuracy and workflow efficiency when integrated into clinical decision pathways. Previous studies have evaluated AI systems against physicians, whereas we aim to compare the benefits of incorporating AI into their final diagnostic decision. This work analyzed the potential for artificial intelligence (AI)-based decision support systems to improve physician accuracy, variability, and efficiency. The decision support system (DSS) assessed was Koios DS, which provides automated sonographic nodule descriptor predictions and a direct cancer risk assessment aligned to ACR TI-RADS. The study was conducted retrospectively between (08/2020) and (10/2020). The set of cases used included 650 patients (21% male, 79% female) of age 53 ± 15. Fifteen physicians assessed each of the cases in the set, both unassisted and aided by the DSS. The order of the reading condition was randomized, and reading blocks were separated by a period of 4 weeks. The system's impact on reader accuracy was measured by comparing the area under the ROC curve (AUC), sensitivity, and specificity of readers with and without the DSS with FNA as ground truth. The impact on reader variability was evaluated using Pearson's correlation coefficient. The impact on efficiency was determined by comparing the average time per read. There was a statistically significant increase in average AUC of 0.083 [0.066, 0.099] and an increase in sensitivity and specificity of 8.4% [5.4%, 11.3%] and 14% [12.5%, 15.5%], respectively, when aided by Koios DS. The average time per case decreased by 23.6% (p = 0.00017), and the observed Pearson's correlation coefficient increased from r = 0.622 to r = 0.876 when aided by Koios DS. These results indicate that providing physicians with automated clinical decision support significantly improved diagnostic accuracy, as measured by AUC, sensitivity, and specificity, and reduced inter-reader variability and interpretation times.

Keywords: Artificial intelligence; Clinical decision support; Diagnostic workflows; TI-RADS; Thyroid ultrasound.

Fig. 1

Imaging presentation and electronic case report form (eCRF) for a benign nodule. A All nodules, regardless of reading condition US or US + DS, are presented to the user with regions of interest predrawn. B The eCRF is presented to the user in one of two default states, from left to right, US alone and US + DS. Each prepopulated descriptor field in the US + DS condition or blank field in the US alone can be selected or updated by clicking on it. In the US + DS condition, the drop down also shows relative probabilities of each of the possible choices generated by the AI system. The AI-based risk adjustment (“Koios Risk Adjustment”) can also be collapsed to remove it from the TI-RADS point total as seen in the far right eCRF example

Fig. 2

Validation dataset descriptive statistics. A Hardware manufacturer distribution stratified by geographical subtype. B Gender distributions for the final validation dataset stratified by location. C Age distributions for the final validation dataset stratified by geographical subtype. D Demographic information for the USA subset, the European Union subset did not have associated demographics available. E Original TI-RADS grading of the nodule assessments originating from the USA as EU nodules did not have TI-RADS assessments

Fig. 3

Performance comparison of US alone to US + DS reading conditions A Per reader AUC shifts represented as the US alone plotted against US + DS. Dashed line represents equivocal performance (y = x); points above the line represent improved AUC for the US + DS condition. B Parametric average ROC curve for all readers on all data for each reading condition. C Per reader operating shifts from US alone (base of the arrow) to US + DS (head of the arrow) when assessed on all data. D Per reader interpretation time comparing US alone to US + DS. Dashed line represents equivocal performance (y = x); points below the line represent faster interpretation for the US + DS condition

Fig. 4

Diagnostic accuracy (measured by AUC) is plotted against interpretation time for US alone A and US + DS B Computing Pearson’s R correlation between performance and interpretation resulted in values of 0.37 and − 0.02 for the US alone and US + DS, respectively. Datapoint colors correspond to the reader legend defined in Fig. 3A