mid-DeepLabv3+: A Novel Approach for Image Semantic Segmentation Applied to African Food Dietary Assessments

Abstract

Recent decades have witnessed the development of vision-based dietary assessment (VBDA) systems. These systems generally consist of three main stages: food image analysis, portion estimation, and nutrient derivation. The effectiveness of the initial step is highly dependent on the use of accurate segmentation and image recognition models and the availability of high-quality training datasets. Food image segmentation still faces various challenges, and most existing research focuses mainly on Asian and Western food images. For this reason, this study is based on food images from sub-Saharan Africa, which pose their own problems, such as inter-class similarity and dishes with mixed-class food. This work focuses on the first stage of VBDAs, where we introduce two notable contributions. Firstly, we propose mid-DeepLabv3+, an enhanced food image segmentation model based on DeepLabv3+ with a ResNet50 backbone. Our approach involves adding a middle layer in the decoder path and SimAM after each extracted backbone feature layer. Secondly, we present CamerFood10, the first food image dataset specifically designed for sub-Saharan African food segmentation. It includes 10 classes of the most consumed food items in Cameroon. On our dataset, mid-DeepLabv3+ outperforms benchmark convolutional neural network models for semantic image segmentation, with an mIoU (mean Intersection over Union) of 65.20%, representing a +10.74% improvement over DeepLabv3+ with the same backbone.

Keywords: CNN; CamerFood10 dataset; food segmentation; semantic segmentation.

Figure 1

Different kinds of Cameroonian food with a similar yellow texture.

Figure 2

Some images in the CamerFood10 dataset with mask overlay.

Figure 3

CamerFood10 class occurrence distribution.

Figure 4

CamerFood10 size distribution of masks from each class based on the number of pixels they occupy in the whole image (i.e., small, medium, large). Small object size < 5% of image; medium object between 5% and 20% of image; large objects > 20% of image.

Figure 5

Architecture of our proposed model: mid-Deeplabv3+.

Figure 6

mid-DeepLabv3+’s feature extraction backbone based on a scaled-down version of the ResNet50 architecture. This is the ResNet50 model without its fifth convolution block (Conv5).

Figure 7

Several images in the CamerFood10 dataset and ground truth mask and prediction with mid-DeepLabv3+ and other benchmark models. For spatial reasons, we only present the predictions of the models with the best results.