Confidence-based laboratory test reduction recommendation algorithm

Abstract

Background: We propose a new deep learning model to identify unnecessary hemoglobin (Hgb) tests for patients admitted to the hospital, which can help reduce health risks and healthcare costs.

Methods: We collected internal patient data from a teaching hospital in Houston and external patient data from the MIMIC III database. The study used a conservative definition of unnecessary laboratory tests, which was defined as stable (i.e., stability) and below the lower normal bound (i.e., normality). Considering that machine learning models may yield less reliable results when trained on noisy inputs containing low-quality information, we estimated prediction confidence to assess the reliability of predicted outcomes. We adopted a "select and predict" design philosophy to maximize prediction performance by selectively considering samples with high prediction confidence for recommendations. Our model accommodated irregularly sampled observational data to make full use of variable correlations (i.e., with other laboratory test values) and temporal dependencies (i.e., previous laboratory tests performed within the same encounter) in selecting candidates for training and prediction.

Results: The proposed model demonstrated remarkable Hgb prediction performance, achieving a normality AUC of 95.89% and a Hgb stability AUC of 95.94%, while recommending a reduction of 9.91% of Hgb tests that were deemed unnecessary. Additionally, the model could generalize well to external patients admitted to another hospital.

Conclusions: This study introduces a novel deep learning model with the potential to significantly reduce healthcare costs and improve patient outcomes by identifying unnecessary laboratory tests for hospitalized patients.

Keywords: Confidence based; Deep learning; Lab test reduction.

Fig. 1

Block diagram of candidate selection. (1) We imputed zeros for missing features before processing the network model. (2) In the training stage, we inserted a random zero mask for existing laboratory values. The network model predicted selection probabilities for individual laboratory tests. Thus, the model ignored some samples, whose inclusions were considered to decrease performance. Each model was trained under one target coverage rate that constrained the actual proportion of selected laboratory tests. Intuitively, the lower coverage rate means that selections are stricter. (3) We chose a model at an acceptable coverage rate in the test stage. A threshold $\tau$ was used to determine whether individual laboratory tests were selected. The selected tests were considered high-confidence candidates. The model recommended canceling pending laboratory tests if predicted values satisfied two joint conditions: a. High confidence; b. Unnecessary (i.e., predicted to be stable and remain normal)

Fig. 2

Model Architecture Framework. In the LSTM network module, the shared LSTM layer received all input features, and outputs hidden features that contained general information derived from original data. The attention-based LSTM layer augmented input embeddings by concatenating hidden features and duplicating original features. One attention-based layer learned a subset of features for the following stability predictor. The other attention-based layer learned entire feature vectors to obtain complicated information for the following normality, value, and selection predictor. In the selective network module, we had four 2-layer MLP predictors to make task-specific predictions for Hgb stability, Hgb normality, Hgb value, and selection probability in parallel. Stability and normality predictors were treated as primary predictions that focus on selected Hgb samples. The value predictor served as the auxiliary prediction that covered all Hgb samples, including the non-selected ones

Fig. 3

Model performance at multiple selection coverages under reduction and no-reduction evaluation. The prevalence refers to the proportion of positive samples (i.e., normal and stable Hgbs) in the selected Hgb candidates. The normality AUC and stability AUC refer to the AUROC for the normality prediction and stability prediction, respectively. The selection threshold $\tau$ is 0.5. The coverage rate refers to the expected proportion of Hgb samples to construct the model

Fig. 4

Consistency between predicted values and predicted normality. The objective was to measure the consistency between predicted values and predicted normality. The “normality accuracy of predicted values” was defined as the percentage of predicted values with $f_V(x_t)>=b$, where $b$ is the value of the LBNR, on Hgb samples with $f_Y(x_t)=1$. We considered a tolerable boundary to be m% lower than the LBNR for predicted normal Hgbs

Fig. 5

Model performances using different selection thresholds for reduction evaluation. The range of our selection threshold is $\tau \in [0.05,0.95]$ with intervals at 0.05

Fig. 6

Comparing model accuracy when evaluating local hospital data and MIMIC III data. The selection threshold $\tau$ is 0.5