How to measure consumer's inconsistency in sensory testing?

Abstract

Consumer sensory testing is the basis for determining directions of product development in the food industry. However, while compliance assessment by trained and expert assessors is well developed, few information is available on testing consumer consistency. Therefore, we provide a statistical framework to rank assessors and attributes according to the level of inconsistency, as well as to identify inconsistent assessors, based on Kendall rank correlation coefficients. The detection of (in)consistency requires evaluations on two connected scales. The suggested approach is illustrated by data from sensory tests of biscuits enriched with three pollens at different levels. 100 consumers evaluated the samples on two different scales (nine category monotonic ascending hedonic response scale, five-category just about right (JAR) intensity scale). The 88 consistent assessors are found using a wider range of both the liking scale and JAR scale than the 12 inconsistent assessors whose evaluations do not have a significantly negative rank correlation. Future consumer tests are recommended to include multiple scales. The proposed framework aims to identify and even filter out the potentially biasing inconsistent evaluations. Questions on attributes leading to highly inconsistent responses should be reconsidered in future sensory tests on the same food product.

Keywords: Consistency; Consumer sensory testing; JAR scale; Liking score; Product development.

Graphical abstract

Fig. 1

Main steps of identifying inconsistent assessors and attributes.

Fig. 2

JAR and liking scores of the least inconsistent assessor (see also Table 5a).

Fig. 3

Histogram of Kendall τ_C values for the 100 assessors.

Fig. 4

Comparison of mean scores for the two types of assessors.

Fig. 5

Comparison of standard deviations for the two types of assessors.

Fig. 6

The relationship between the standard deviation of scores and Kendall τ_C coefficients for the assessors. Notes: Blue points represent consistent assessors. Red points represent inconsistent assessors.

Fig. 7

The relationship between the number of concordant and discordant evaluation pairs and Kendall τ_C coefficients for the assessors. Notes: Blue points represent consistent assessors. Red points represent inconsistent assessors. The dashed line shows the linear trend, which is y = 0.3459 − 0.0003x (R² = 0.2861).

Fig. 8

The relationship between the standard deviation of scores and Kendall τ_C coefficients for the attributes. Notes: Each point represents an attribute. The dashed line shows the linear trend, which is y = 0.1566 − 0.2632x (R² = 0.3523) for Fig. 8a and y = 0.4312 − 0.8502x (R² = 0.2138) Fig. 8b. The abbreviations of the attributes are given in Table 6.

Fig. 9

The relationship between global taste and global liking scores. Notes: The ratio of consistent and inconsistent assessors is shown by blue and orange circular sectors. The three lines shows linear trends, which are y = 1.21097 + 0.76649x (R² = 0.5619) for consistent assessors (dashed blue); y = 2.53432 + 0.58105x (R² = 0.3428) for inconsistent assessors (solid red); y = 1.34029 + 0.74841x (R² = 0.5381) for all assessors (black dotted).