Does the number of choice sets matter? Results from a web survey applying a discrete choice experiment

Abstract

Optimising the design of discrete choice experiments (DCE) involves maximising not only the statistical efficiency, but also how the nature and complexity of the experiment itself affects model parameters and variance. The present paper contributes by investigating the impact of the number of DCE choice sets presented to each respondent on response rate, self-reported choice certainty, perceived choice difficulty, willingness-to-pay (WTP) estimates, and response variance. A sample of 1053 respondents was exposed to 5, 9 or 17 choice sets in a DCE eliciting preferences for dental services. Our results showed no differences in response rates and no systematic differences in the respondents' self-reported perception of the uncertainty of their DCE answers. There were some differences in WTP estimates suggesting that estimated preferences are to some extent context-dependent, but no differences in standard deviations for WTP estimates or goodness-of-fit statistics. Respondents exposed to 17 choice sets had somewhat higher response variance compared to those exposed to 5 choice sets, indicating that cognitive burden may increase with the number of choice sets beyond a certain threshold. Overall, our results suggest that respondents are capable of managing multiple choice sets - in this case 17 choice sets - without problems.