goFOODTM: An Artificial Intelligence System for Dietary Assessment

Abstract

Accurate estimation of nutritional information may lead to healthier diets and better clinical outcomes. We propose a dietary assessment system based on artificial intelligence (AI), named goFOOD^TM. The system can estimate the calorie and macronutrient content of a meal, on the sole basis of food images captured by a smartphone. goFOOD^TM requires an input of two meal images or a short video. For conventional single-camera smartphones, the images must be captured from two different viewing angles; smartphones equipped with two rear cameras require only a single press of the shutter button. The deep neural networks are used to process the two images and implements food detection, segmentation and recognition, while a 3D reconstruction algorithm estimates the food's volume. Each meal's calorie and macronutrient content is calculated from the food category, volume and the nutrient database. goFOOD^TM supports 319 fine-grained food categories, and has been validated on two multimedia databases that contain non-standardized and fast food meals. The experimental results demonstrate that goFOOD^TM performed better than experienced dietitians on the non-standardized meal database, and was comparable to them on the fast food database. goFOOD^TM provides a simple and efficient solution to the end-user for dietary assessment.

Keywords: calorie; carbohydrate; computer vision; fat; nutrient estimation; protein; smartphone.

Figure 1

Overview of goFOOD^TM.

Figure 2

The application: (a) goFOOD^TMLite—two images capturing (b) goFOOD^TMLite—Video recording.

Figure 3

The application: (a) goFOOD^TM—Successful automatic segmentation; (b) goFOOD^TM— Failed automatic segmentation due to bad lighting [left]—Manual user input [middle]— Successful semi-automatic segmentation [right]; (c) goFOOD^TM—Automatic Recognition.

Figure 4

The food categories are organized in a three-level hierarchy. The green labels indicate fine-grained food categories supported by the system, while the gray and blue labels are the concluded first and second level hyper food categories, respectively.

Figure 5

Some example meal images in (a) MADiMa and (b) Fast food Databases.

Figure 6

Examples of correctly and incorrectly recognized food images.

Figure 7

Bland–Altman plots of goFOOD^TM’s and dietitians’ estimations on the MADiMa database in terms of (a) CHO, (b) PRO, (c) FAT and (d) Calories. The dashed lines indicate the $95\%$ confidence interval of goFOOD^TM (blue) and the dietitians’ estimations (red).

Figure 8

Bland–Altman plots of goFOOD^TM’s and dietitians’ estimations on the Fast Food database in terms of (a) CHO, (b) PRO, (c) FAT and (d) Calories. The dashed lines indicate the $95\%$ confidence interval of goFOOD^TM (blue) and the dietitians’ estimations (red).