Evaluation of a Novel Artificial Intelligence System to Monitor and Assess Energy and Macronutrient Intake in Hospitalised Older Patients

Abstract

Malnutrition is common, especially among older, hospitalised patients, and is associated with higher mortality, longer hospitalisation stays, infections, and loss of muscle mass. It is therefore of utmost importance to employ a proper method for dietary assessment that can be used for the identification and management of malnourished hospitalised patients. In this study, we propose an automated Artificial Intelligence (AI)-based system that receives input images of the meals before and after their consumption and is able to estimate the patient's energy, carbohydrate, protein, fat, and fatty acids intake. The system jointly segments the images into the different food components and plate types, estimates the volume of each component before and after consumption, and calculates the energy and macronutrient intake for every meal, based on the kitchen's menu database. Data acquired from an acute geriatric hospital as well as from our previous study were used for the fine-tuning and evaluation of the system. The results from both our system and the hospital's standard procedure were compared to the estimations of experts. Agreement was better with the system, suggesting that it has the potential to replace standard clinical procedures with a positive impact on time spent directly with the patients.

Keywords: artificial intelligence; dietary assessment; dietary intake; geriatrics; malnutrition.

Figure 1

Standardised mount for capturing images by the RGB-D camera; (a) setup for image capture—the RGB-D camera is connected to a laptop and on top of an empty tray; (b) expanded view of the mount showing the mechanical parts of the mount for the RGB-D camera.

Figure 2

Sample images before and after meal consumption from Geriatrische Klinik St. Gallen (a,b) and from the Bern University Hospital [24] (c,d).

Figure 3

The plates used in the Geriatrische Klinik, St. Gallen: (a) round plate for main course; (b) bowl for soup; (c) square bowl for salad or dessert; (d) glass for dessert.

Figure 4

An example of exported pdf file from the kitchen database for the dish “Broccoli”: (a) the quantity of a normal-sized dish; (b) the macronutrients of a normal-sized dish (P: protein, F: fat, FA: fatty acids, CHO: carbohydrates).

Figure 5

The segmentation tool that was used to provide the ground truth of the segmentation (GT_seg): (a) the interface of the segmentation tool; (b) the semi-automatic segmentation; (c) the segmented plates of the images (up) and food types (bottom).

Figure 6

The system receives as input the daily menu, the RGB-D images, and the plate and meal segmentation masks and estimates the volume of each dish before and after consumption.

Figure 7

Segmentation results using different architectures: (a) the original RGB image; (b) the GT segmentation mask (GT_seg); (c) the segmentation mask from the Encoder + PSPNet; (d) the segmentation mask from DeepLabv3; (e) the segmentation mask from ResNet + PSPNet w/pretraining.

Figure 8

Bar plots for the 20 testing meals ordered by consumed percentage of each meal. (a) Schematic depiction of the workflow. Estimations for the (b) energy (kcal); (c) CHO (g); (d) protein (g); (e) fat (g); (f) fatty acids (g) intake. The green bars indicate the visual estimations of the dietitians and the student (reference), the blue bars indicate the system’s estimations, and the red bars the nursing staff following the standard clinical procedure.

Figure 9

Differences in calories and macronutrient estimations per meal for the estimation methods. Difference was calculated by subtraction of the estimated calories or macronutrients of the system or the standard clinical procedure (SCP) from the visual estimation (reference) for each meal. Variation between the different estimations depicted as a box-plot in (a), where the circle and the square indicate the outliers, or as paired individual values in (b). The visual estimation is shown as a dotted line at a value of 0. Statistical analysis was performed using a paired-t-test. * = p ≤ 0.05.