A Multisensor Data Fusion Approach for Predicting Consumer Acceptance of Food Products

Abstract

Sensory experiences play an important role in consumer response, purchase decision, and fidelity towards food products. Consumer studies when launching new food products must incorporate physiological response assessment to be more precise and, thus, increase their chances of success in the market. This paper introduces a novel sensory analysis system that incorporates facial emotion recognition (FER), galvanic skin response (GSR), and cardiac pulse to determine consumer acceptance of food samples. Taste and smell experiments were conducted with 120 participants recording facial images, biometric signals, and reported liking when trying a set of pleasant and unpleasant flavors and odors. Data fusion and analysis by machine learning models allow predicting the acceptance elicited by the samples. Results confirm that FER alone is not sufficient to determine consumers' acceptance. However, when combined with GSR and, to a lesser extent, with pulse signals, acceptance prediction can be improved. This research targets predicting consumer's acceptance without the continuous use of liking scores. In addition, the findings of this work may be used to explore the relationships between facial expressions and physiological reactions for non-rational decision-making when interacting with new food products.

Keywords: consumer acceptance prediction; data fusion; emotion recognition; facial expression recognition; galvanic skin response; machine learning; neural networks; sensory analysis.

Figure 1

Sweet gum elaboration process: (a) gelatin in water, (b) sugar, water, and glucose mixture, (c) gelatin solution, color, flavor, and citric acid in the mixture, (d) bed of starch, (e) mixture in the bed of starch, and (f) resulting sweet gums.

Figure 2

Odor sample: (a) Soaking cotton with solution. (b) Wooden stick in container.

Figure 3

Booth setup.

Figure 4

The sensory analysis system architecture. GSR, galvanic skin response.

Figure 5

Schematic representation of a three-input perceptron.

Figure 6

Two-layer neural network architecture. For the sake of simplicity and clarity, individual weights are not shown.

Figure 7

Numbered facial landmarks.

Figure 8

Neural network architecture for the first stage.

Figure 9

Example of detected emotion probabilities.

Figure 10

Decision tree example: to predict the consumer’s evaluation, questions need to be answered from top to bottom and following the path of the answers. At the end of the path, the last node contains the prediction of the consumer’s evaluation.

Figure 11

Acceptance results for taste evaluations.

Figure 12

Acceptance results for smell evaluations.

Figure 13

Recognized emotions: (a) taste and (b) smell experiments.

Figure 14

Correlation matrix of facial emotion recognition (FER), sensor responses, and consumer acceptance: (a,c) display the correlation matrices of taste experiments, while (b,d) show the correlation matrices of smell experiments. Cells contain correlations between column and row features.

Figure 15

Feature importance: (a) taste and (b) smell regression models.

Figure 16

GSR and pulse measurements for the taste tests: (a) The participants’ GSR average values and (b) standard deviation. (c) The participants’ pulse average values and (d) standard deviation.

Figure 17

GSR and pulse measurements for the smell tests. (a) The participants’ GSR average values and (b) standard deviation. (c) The participants’ pulse average values and (d) standard deviation.