Assessing respondent-driven sampling

Abstract

Respondent-driven sampling (RDS) is a network-based technique for estimating traits in hard-to-reach populations, for example, the prevalence of HIV among drug injectors. In recent years RDS has been used in more than 120 studies in more than 20 countries and by leading public health organizations, including the Centers for Disease Control and Prevention in the United States. Despite the widespread use and growing popularity of RDS, there has been little empirical validation of the methodology. Here we investigate the performance of RDS by simulating sampling from 85 known, network populations. Across a variety of traits we find that RDS is substantially less accurate than generally acknowledged and that reported RDS confidence intervals are misleadingly narrow. Moreover, because we model a best-case scenario in which the theoretical RDS sampling assumptions hold exactly, it is unlikely that RDS performs any better in practice than in our simulations. Notably, the poor performance of RDS is driven not by the bias but by the high variance of estimates, a possibility that had been largely overlooked in the RDS literature. Given the consistency of our results across networks and our generous sampling conditions, we conclude that RDS as currently practiced may not be suitable for key aspects of public health surveillance where it is now extensively applied.

Fig. 1.

Design effects for the 13 binary traits in Project 90 (A) and the 46 binary traits in Add Health (B); in Add Health, each circle indicates the design effect of a trait in one of 84 schools.

Fig. 2.

Coverage rates of nominal 95% RDS confidence intervals in Project 90 (A) and Add Health (B). For each trait in Add Health, the mean coverage over all 84 schools is shown.

Fig. 3.

Comparison of bias, standard error, and RMSE between the RDS estimator and the unweighted mean of the RDS sample in Project 90 (A–C) and Add Health (D–F). Each point corresponds to a given trait in a given network population.