Applying queueing theory to evaluate wait-time-savings of triage algorithms

Abstract

In the past decade, artificial intelligence (AI) algorithms have made promising impacts in many areas of healthcare. One application is AI-enabled prioritization software known as computer-aided triage and notification (CADt). This type of software as a medical device is intended to prioritize reviews of radiological images with time-sensitive findings, thus shortening the waiting time for patients with these findings. While many CADt devices have been deployed into clinical workflows and have been shown to improve patient treatment and clinical outcomes, quantitative methods to evaluate the wait-time-savings from their deployment are not yet available. In this paper, we apply queueing theory methods to evaluate the wait-time-savings of a CADt by calculating the average waiting time per patient image without and with a CADt device being deployed. We study two workflow models with one or multiple radiologists (servers) for a range of AI diagnostic performances, radiologist's reading rates, and patient image (customer) arrival rates. To evaluate the time-saving performance of a CADt, we use the difference in the mean waiting time between the diseased patient images in the with-CADt scenario and that in the without-CADt scenario as our performance metric. As part of this effort, we have developed and also share a software tool to simulate the radiology workflow around medical image interpretation, to verify theoretical results, and to provide confidence intervals for the performance metric we defined. We show quantitatively that a CADt triage device is more effective in a busy, short-staffed reading setting, which is consistent with our clinical intuition and simulation results. Although this work is motivated by the need for evaluating CADt devices, the evaluation methodology presented in this paper can be applied to assess the time-saving performance of other types of algorithms that prioritize a subset of customers based on binary outputs.

Fig. 1

Radiologist workflows without and with a CADt device. Top: the without-CADt scenario in which patient images are reviewed in the order of their arrival. Bottom: the with-CADt workflow in which AI-positive patient images are reviewed first before the AI-negative images. In both scenarios, the radiologist may be interrupted by interrupting cases. All cartoon icons are adopted from Microsoft PowerPoint application

Fig. 2

Priority classes in two scenarios. Left: Without-CADt scenario has two priority classes. Interrupting patient images with an arrival rate of ${\lambda }_f$ have a higher priority, whereas non-interrupting patient images (${\lambda }_{\text{non}f}$) have a lower priority. Within the non-interrupting priority class is a mix of diseased and non-diseased patient images. Right: The with-CADt scenario has three priority classes. Interrupting patient images with an arrival rate of ${\lambda }_f$ have the highest priority, AI-positive patient images (${\lambda }_{+}$) have a middle priority, and AI-negative patient images (${\lambda }_{-}$) have the lowest priority. Within the AI-positive class, there is a mix of diseased patient images (i.e., true-positive) and non-diseased patient images (i.e., false-positive). Similarly, AI-negative cases consist of diseased (false-negative) patient images and non-diseased (true-negative) patient images. The parameters associated with the arrows represent the conditional probabilities that a patient case belongs to a subgroup. For example, in the without CADt scenario, $\pi$ represents the probability that a non-interrupting case belongs to the diseased subgroup. See Sect. 2.1 for the definitions of the parameters

Fig. 3

The RDR-truncated transition diagram for non-interrupting patient images in Model 1 without a CADt device. The states $(n_f,n_{\text{non}f})$ are defined by the numbers of interrupting and non-interrupting cases in the system

Fig. 4

An Erlang–Coxian (EC) distribution with one Erlang phase and two Coxian phases. See [11] for the closed form solutions to calculate these parameters

Fig. 5

The RDR-truncated transition diagram for AI-negative patient images in Model 1 with a CADt device. The state is defined as $(n_f,n_{+},n_{-})$, and states with $n_f+n_{+}\ge 2$ are truncated. A total of 6 busy periods are identified. Each busy period i has a transition rate ${\mathbf{B}}_i$ along with a conditional probability that it ends at a certain state given that it starts with a particular state

Fig. 6

The RDR-truncated transition diagram for non-interrupting patient images in Model 2 in a without-CADt scenario. The state is defined by the number of interrupting patient images (either 0 or $1^{+}$), number of non-interrupting patient images n, and the disease status of case that the one radiologist is reviewing (either D for diseased or ND for non-diseased). The “$\to D$" and “$\to ND$" in the truncated states keep track of the disease status of the interrupted case

Fig. 7

The RDR-truncated transition diagram for AI-negative subgroup in Model 2 with a CADt device. State is defined as $(n_f,n_{+},i,n_{-},j)$, where i (or j) indicates the disease status of the AI-positive (or AI-negative) case the radiologist is reading

Fig. 8

Amount of time saved per diseased patient image as a function of (top) traffic intensity, (middle) disease prevalence, and (bottom) interrupting fraction. Green and blue lines represent scenarios with one and two radiologists, respectively. Dashed lines are theoretical $\delta W_D$, and the solid lines represent the mean wait-time-savings from simulation. Shaded areas are the 95% confidence intervals (C.I.s) from simulation. The average reading time ($1/{\mu }_f$) for an interrupting image is set at 5 min, whereas the average reading time for the diseased ($1/{\mu }_D$) and non-diseased ($1/{\mu }_{\mathit{ND}}$) subgroups are both 10 min

Fig. 9

Amount of time saved per diseased patient as a function of (top) traffic intensity, (middle) disease prevalence, and (bottom) interrupting fraction. Only one radiologist is on-site, and its average reading times for interrupting ($1/{\mu }_f$) and diseased ($1/{\mu }_D$) patient images are set at 5 min and 10 min, respectively. The average reading time for non-diseased ($1/{\mu }_{\mathit{ND}}$) patient images varies between 5 min (orange), 10 min (green), and 15 min (red). Dashed lines are theoretical $\delta W_D$, and the solid lines represent the mean wait-time-savings from simulation. Shaded areas are the 95% confidence intervals (C.I.s) from simulation. Note that the green set of lines here is identical to that in Fig. 8

Fig. 10

A summary ROC plot for evaluating both the diagnostic and wait-time-savings of a CADt device. The middle rainbow plot is an ROC space with an ROC curve (dashed dark gray) of a theoretical CADt device. Color map represents theoretical mean time savings ($\delta W_D$) per diseased patient image, assuming a disease prevalence of 10% in a relatively busy hospital (traffic intensity of 0.8) with only one radiologist and no interrupting patient images. The radiologist’s average reading times for diseased ($1/{\mu }_D$) and non-diseased ($1/{\mu }_{\mathit{ND}}$) patient images are both set at 10 min. Positive $\delta W_D$ (blue region) means an overall time delay for diseased patient images, and negative $\delta W_D$ (red region) means an overall time savings. The values printed on the color map are the $\delta W_D$’s at the corresponding points of false-positive and true-positive rates. The dot represents the pre-determined AI operating point ($\text{Se}=95\%,\text{Sp}=89\%$). Top plot represents the theoretical $\delta W_D$ (dashed gray) along the ROC curve as a function of false-positive rate, and left plot represents the same theoretical $\delta W_D$ (dashed gray) along the curve but as a function of true-positive rate. The green solid line represents the mean time savings along the ROC curve obtained from simulation. The darker and lighter shaded areas indicate the 68% and 95% ranges from simulation around the mean time savings. The black dotted vertical and horizontal lines indicate that the theoretical mean time-saving for diseased patients is roughly 36 min at the given operating point. The color axis is translated to stroke patient outcome metrics based on Table 12 in Supplementary Content (Supplementary 2) of [12]

Fig. 11

The transition diagram to calculate inter-level passage times within the AI-positive class (middle priority) in Model 1 in a with-CADt scenario

Fig. 12

The transition diagram to calculate inter-level passage time within the AI-positive class in Model 2 in a with-CADt scenario