ANN uncertainty estimates in assessing fatty liver content from ultrasound data

Abstract

Background and objective: This article uses three different probabilistic convolutional architectures applied to ultrasound image analysis for grading Fatty Liver Content (FLC) in Metabolic Dysfunction Associated Steatotic Liver Disease (MASLD) patients. Steatosis is a new silent epidemic and its accurate measurement is an impelling clinical need, not only for hepatologists, but also for experts in metabolic and cardiovascular diseases. This paper aims to provide a robust comparison between different uncertainty quantification strategies to identify advantages and drawbacks in a real clinical setting.

Methods: We used a classical Convolutional Neural Network, a Monte Carlo Dropout, and a Bayesian Convolutional Neural Network with the goal of not only comparing the goodness of the predictions, but also to have access to an evaluation of the uncertainty associated with the outputs.

Results: We found that even if the prediction based on a single ultrasound view is reliable (relative RMSE [5.93%-12.04%]), networks based on two ultrasound views outperform them (relative RMSE [5.35%-5.87%]). In addition, the results show that the introduction of a "not confident" category contributes to increase the percentage of correctly predicted cases and to decrease the percentage of mispredicted cases, especially for semi-intrusive methods.

Conclusions: The possibility of having access to information about the confidence with which the network produces its outputs is a great advantage, both from the point of view of physicians who want to use neural networks as computer-aided diagnosis, and for developers who want to limit overfitting and obtain information about dataset problems in terms of out-of-distribution detection.

Keywords: Artificial neural networks; Bayesian uncertainty; Fatty liver content; Ultrasound imaging; Uncertainty quantification.

Graphical abstract

Fig. 1

Pre-processing steps: (a) a water-shed with jumps algorithm identifies and removes annotations on the original image; (b) the ultrasound cone is identified and the image is cropped; (c) the cone is transformed into polar coordinates.

Fig. 2

Scheme of the three models that are used for the evaluation of the degree of steatosis from the AR and HR images. The standard model (deterministic CNN) takes as input AR and HR images and, after a pre-processing phase described in Section 2.2, it applies: a batch normalization, 6 convolutional layers (2,4,8,16,8,4) followed by an average pooling. The AR and HR are pre-trained and then combined (flatten layer followed by a dense one) which is fine-tuned to produce a deterministic prediction (regression). The other two models (MC-Dropout and Bayesian Neural Network) have the same structure and hyperparameters, but provide two outputs: a prediction (regression) and a reliability score. The MC-Dropout includes a dropout layer (p=10%) after each average pooling and before the combined dense layer, and during inference uses these layers to define several predictions whose variability is a measure of reliability. Instead, the Bayesian model has two outputs instead of one (mean and standard deviation), which when combined with an ad hoc loss, allows both a prediction and the corresponding reliability score to be trained directly.

Fig. 3

Bland Altman plots of the three different models against the ground truth (GT). The graph shows the mean difference and the 95% CI.

Fig. 4

Confusion matrices of the three architectures used. a) Classical CNN Confusion matrix. b) e) Confusion matrices of the MC Dropout, respectively without and with the “Non Confident” category. c) f) Confusion matrices of the Bayesian CNN, respectively without and with “Non Confident” category. d) Legenda for e) and f) Figures: all outputs for which the confidence band overlaps the value of 3.12 (threshold value of fat percentage between healthy and pathological subjects) are defined “Non Confident”. All outputs for which the confidence band is entirely below the value of 3.12 are defined as Healthy. The outputs for which the confidence band is entirely above the value of 3.12 are defined as Pathological.

Fig. 5

Comparison of predictions and level of uncertainty for 4 cases with different percentage of FLC. a) Case predicted with low precision and small amount of uncertainty. b) Case predicted with good precision and big amount of uncertainty. c) Case predicted with low precision and big amount of uncertainty. d) Case predicted with good precision and small amount of uncertainty.