"Snap-n-Eat": Food Recognition and Nutrition Estimation on a Smartphone

Abstract

We present snap-n-eat, a mobile food recognition system. The system can recognize food and estimate the calorific and nutrition content of foods automatically without any user intervention. To identify food items, the user simply snaps a photo of the food plate. The system detects the salient region, crops its image, and subtracts the background accordingly. Hierarchical segmentation is performed to segment the image into regions. We then extract features at different locations and scales and classify these regions into different kinds of foods using a linear support vector machine classifier. In addition, the system determines the portion size which is then used to estimate the calorific and nutrition content of the food present on the plate. Previous approaches have mostly worked with either images captured in a lab setting, or they require additional user input (eg, user crop bounding boxes). Our system achieves automatic food detection and recognition in real-life settings containing cluttered backgrounds. When multiple food items appear in an image, our system can identify them and estimate their portion size simultaneously. We implemented this system as both an Android smartphone application and as a web service. In our experiments, we have achieved above 85% accuracy when detecting 15 different kinds of foods.

Keywords: food recognition; mobile food recognition; nutrition estimation; visual food recognition.

Figure 1.

Overview of proposed approach.

Figure 2.

We computed the low level features inside labeled food items. The top 2 images show the embedding of the HOG feature. The bottom 2 images show the embedding of SIFT feature at scales of 16 × 16 and 32 × 32. Each colored dot represents 1 labeled food instance.

Figure 3.

Saliency computation. The left image shows a picture of a food plate snapped by a user. The center image shows the saliency image with red/yellow colors indicating high saliency and blue colors indicating a low saliency. The right image shows the binary saliency map obtained by applying a threshold on the center image which can be used to identify the location of food in the image. Note that the binary saliency map picks out the most important regions of the image.

Figure 4.

Sampled segmentation. The figure shows an example of our segmentation. The food items present in a food plate are segmented into different regions, each of which usually corresponds to a single food item. The regions in green correspond to the steak, the regions in blue correspond to fruit, and the regions in red correspond to the roll. Classifiers are then run on each of these regions.

Figure 5.

Semantic ambiguities in food items. Left: 4 food items are assigned the same label “sandwiches.” Right: 4 visually similar items are assigned different labels.

Figure 6.

Food ontology. An example of a food ontology with an enlarged view of a small section.

Figure 7.

Examples of nutrition estimation on the test data set. In each image the food items and the detected class label below it have been given the same color.