Utilizing machine learning to predict participant response to follow-up health surveys in the Millennium Cohort Study

Abstract

The Millennium Cohort Study is a longitudinal study which collects self-reported data from surveys to examine the long-term effects of military service. Participant nonresponse to follow-up surveys presents a potential threat to the validity and generalizability of study findings. In recent years, predictive analytics has emerged as a promising tool to identify predictors of nonresponse. Here, we develop a high-skill classifier using machine learning techniques to predict participant response to follow-up surveys of the Millennium Cohort Study. Six supervised algorithms were employed to predict response to the 2021 follow-up survey. Using latent class analysis (LCA), we classified participants based on historical survey response and compared prediction performance with and without this variable. Feature analysis was subsequently conducted on the best-performing model. Including the LCA variable in the machine learning analysis, all six algorithms performed comparably. Without the LCA variable, random forest outperformed the benchmark regression model, however overall prediction performance decreased. Feature analysis showed the LCA variable as the most important predictor. Our findings highlight the importance of historical response to improve prediction performance of participant response to follow-up surveys. Machine learning algorithms can be especially valuable when historical data are not available. Implementing these methods in longitudinal studies can enhance outreach efforts by strategically targeting participants, ultimately boosting survey response rates and mitigating nonresponse.

Keywords: Latent class analysis; Longitudinal studies; Machine learning; Survey nonresponse; Survey outreach efforts; Survey response.

Fig. 1

Temporal prequential approach, using a sliding window schematic, to train and test supervised machine learning models using Panel 1 data from the Millennium Cohort Study. Survey response was predicted using predictor variables from the prior survey cycle in both the train and test data.

Fig. 2

Comparison of prediction performance using six supervised machine learning algorithms on the 2016–2021 test data, employing the receiver operating characteristic and precision-recall curves. Panels (a) and (b) demonstrate the results when latent class analysis (LCA) is incorporated, while panels (c) and (d) show the outcomes without LCA. The dotted lines represent the performance at random chance levels. Prediction performance among the six algorithms exhibited comparable results with the inclusion of the LCA variable, whereas the random forest and interaction forest algorithms displayed noticeably higher area under the curve values compared with the other curves when the LCA variable was omitted.

Fig. 3

Feature importance of predictor variables in the random forest model, both with and without the incorporation of latent class analysis (LCA), utilizing the 2016–2021 test data. In panel (a), where LCA is used, the model demonstrates a pronounced preference for the LCA variable, showcasing its crucial role. However, in panel (b), without LCA, the model identifies a distinct set of patterns in the data, resulting in a noticeable shift in the overall importance of the variables. AD, active duty; GWV, Gulf War veteran.