Incorporating Level of Effort Paradata in the NSDUH Nonresponse Adjustment Process [Internet]

Excerpt

Background: In 2014 and 2015, the National Survey on Drug Use and Health (NSDUH) underwent changes to its sampling methodology, questionnaire content, and data collection procedures. This provided an opportunity to investigate new approaches to nonresponse adjustment that may improve overall NSDUH data quality in the face of declining response rates. This report describes the results of an investigation of the callback modeling approach for adjusting for nonresponse. Callback modeling uses data on respondent availability and contact difficulty to adjust for nonignorable nonresponse bias.

Method: This report uses NSDUH record of call data provided by field interviewers and the final case disposition to assess the potential viability and impact of adjusting for these characteristics. Two basic approaches for incorporating this information into the nonresponse adjustment process were investigated in this study. One approach, the callback model, essentially equates a person's response propensity to the product of the probability of contact at each attempt and the probability of an interview, given that an initial contact was made. The second approach (referred to as GEM+) incorporates a level of effort (LOE) variable in a traditional logistic regression model for response propensity. Both approaches were compared with the current logistic regression approach (referred to as GEM).

Results: The best callback model did not outperform either of the regression models. Improving model fit did not always improve model performance. In fact, specifying more complex models actually increased model bias in some cases even while model fit significantly improved. These results suggest that the lackluster performance of the callback modeling approach in this study is attributable to callback data errors.

Conclusion: Although the number of callbacks required to establish contact with a sample person is strongly correlated with that person's contactability, including callbacks in the modeling may improve nonresponse bias if the quality of the callback data were higher, but the improvements may not be worth the LOE required to improve data quality or implement the new procedures.