"KAIZEN" method realizing implementation of deep-learning models for COVID-19 CT diagnosis in real world hospitals

Abstract

Numerous COVID-19 diagnostic imaging Artificial Intelligence (AI) studies exist. However, none of their models were of potential clinical use, primarily owing to methodological defects and the lack of implementation considerations for inference. In this study, all development processes of the deep-learning models are performed based on strict criteria of the "KAIZEN checklist", which is proposed based on previous AI development guidelines to overcome the deficiencies mentioned above. We develop and evaluate two binary-classification deep-learning models to triage COVID-19: a slice model examining a Computed Tomography (CT) slice to find COVID-19 lesions; a series model examining a series of CT images to find an infected patient. We collected 2,400,200 CT slices from twelve emergency centers in Japan. Area Under Curve (AUC) and accuracy were calculated for classification performance. The inference time of the system that includes these two models were measured. For validation data, the slice and series models recognized COVID-19 with AUCs and accuracies of 0.989 and 0.982, 95.9% and 93.0% respectively. For test data, the models' AUCs and accuracies were 0.958 and 0.953, 90.0% and 91.4% respectively. The average inference time per case was 2.83 s. Our deep-learning system realizes accuracy and inference speed high enough for practical use. The systems have already been implemented in four hospitals and eight are under progression. We released an application software and implementation code for free in a highly usable state to allow its use in Japan and globally.

Figure 1

Visual abstract. The overview of our work is described in this figure. A large number of CT images were collected and labeled by eight radiologists. Two binary classification models were trained and evaluated by these image datasets. An inference program to execute these models was constructed and implemented to real-world hospitals. All of the process was conducted based on the “KAIZEN checklist”.

Figure 2

Flowchart of the process for inclusion and exclusion of the collected patients’ data. After exclusion, only CT images at the initial imaging of each patient are included in the slice model. All CT series, including follow-up, are included in the series model.

Figure 3

AI system architecture. Raw DICOM images are standardized and molded in the pre-processing unit. These images are input into each of the models in the diagnostic model unit to output probability scores for COVID-19.

Figure 4

ROC curves of the slice and series models. The ROC curves of the slice and series models for the validation and test data are shown in Fig. 3. The AUC values and their 95% confidence intervals are also shown.

Figure 5

Flowchart of our inference process. Dashed arrows indicate the use of outputs in the past steps.

Figure 6

Hospitals that implement our AI system.