A Comprehensive Survey of Image-Based Food Recognition and Volume Estimation Methods for Dietary Assessment

Abstract

Dietary studies showed that dietary problems such as obesity are associated with other chronic diseases, including hypertension, irregular blood sugar levels, and increased risk of heart attacks. The primary cause of these problems is poor lifestyle choices and unhealthy dietary habits, which are manageable using interactive mHealth apps. However, traditional dietary monitoring systems using manual food logging suffer from imprecision, underreporting, time consumption, and low adherence. Recent dietary monitoring systems tackle these challenges by automatic assessment of dietary intake through machine learning methods. This survey discusses the best-performing methodologies that have been developed so far for automatic food recognition and volume estimation. Firstly, the paper presented the rationale of visual-based methods for food recognition. Then, the core of the study is the presentation, discussion, and evaluation of these methods based on popular food image databases. In this context, this study discusses the mobile applications that are implementing these methods for automatic food logging. Our findings indicate that around 66.7% of surveyed studies use visual features from deep neural networks for food recognition. Similarly, all surveyed studies employed a variant of convolutional neural networks (CNN) for ingredient recognition due to recent research interest. Finally, this survey ends with a discussion of potential applications of food image analysis, existing research gaps, and open issues of this research area. Learning from unlabeled image datasets in an unsupervised manner, catastrophic forgetting during continual learning, and improving model transparency using explainable AI are potential areas of interest for future studies.

Keywords: automatic diet monitoring; feature extraction; food datasets; food recognition; image analysis; interactive segmentation; volume estimation.

Figure 1

Scope and taxonomy of this survey paper.

Figure 2

Confusion matrix.

Figure 3

System Flow.

Figure 4

Sample images from few food datasets.

Figure 5

Handcrafted feature extraction methods.

Figure 6

Food Volume Estimation Methods.

Figure 7

Strengths and weaknesses of automatic food estimation methods.

Figure 8

The application provides the top prediction result. This picture is taken from the study of Ghalib et al., 2020 (permission has been obtained from original author).

Figure 9

Percentage of datasets summarized according to the types of food. Generic refers to the multi-cultural dataset.

Figure 10

Percentage of studies summarized according to the type of feature extraction methods.

Figure 11

Volume estimation methods using single images vs. multiple images.

Figure 12

Percentage of studies summarized according to the type of methods employed for feature extraction from food images and the category of classifier used for food image analysis in a mobile application.