Comparing ChatGPT and validated questionnaires in assessing loneliness and online social support among college students: a cross-sectional study

Abstract

The capability of ChatGPT to understand and generate human-readable text has prompted the investigation of its potential as mental health assessment tools. This study aims to explore the validity of ChatGPT in assessing loneliness and online social support among college students by comparing scoring consistency between ChatGPT and the validated questionnaires. This was a cross-sectional study between June and August 2024. We pre-trained ChatGPT-4 based on the validated University of California Los Angeles Loneliness Scale-6 (ULS-6) and Chinese Youth Version of the Online Social Support Scale (OSSS-CS), creating a structured interview questionnaire. Participants were invited to complete both the ChatGPT-created questionnaire and the validated questionnaires. We used Spearman correlation analysis, Intra-class correlation coefficients (ICC), and Bland-Altman plots to assess the agreement between the scores from ChatGPT and the validated questionnaires. In addition, we evaluated ceiling and floor effects. A total of 216 college students participated the survey. The results demonstrated a good consistency between the scores obtained from ChatGPT and the validated questionnaires, with ICC of 0.81 (95% CI 0.75-0.85, p < 0.001) for ULS-6 and 0.95 (95% CI 0.94-0.96, p < 0.001) for OSSS-CS. The Spearman correlation coefficients were 0.64 (p < 0.001) for ULS-6 and 0.89 (p < 0.001) for OSSS-CS, indicating a moderate correlation. No ceiling or floor effects were observed. The ChatGPT-created questionnaire demonstrated acceptable consistency with the validated questionnaires. Future studies can further explore the performance of ChatGPT in different populations and domains, as well as how to integrate it with validated questionnaires to enhance the accessibility of assessments.

Keywords: Artificial intelligence; ChatGPT; Large language models; Loneliness; Online social support; Questionnaires.

Fig. 1

Spearman correlation matrix of the ChatGPT questionnaire and the validated loneliness and online social support questionnaires scores: the heatmap visualizes the Spearman correlation matrix for the ChatGPT questionnaire and the validated questionnaires scores. Heatmap (a) compares the Spearman correlation coefficients and the degree of correlation between GPT-ULS-6 and ULS-6; Heatmap (b) compares the Spearman correlation coefficients and the degree of correlation between GPT-OSSS-CS and OSSS-CS. Each cell in the heatmap represents the Spearman correlation coefficient between two variables, with the color intensity and the value indicating the strength and direction of the correlation. Asterisks indicates significant values: *p < 0.05, **p < 0.01, ***p < 0.001.

Fig. 2

(a) Bland–Altman plot of difference between GPT-ULS-6 and ULS-6 change total scores (n = 216). (b) Bland–Altman plot of difference between GPT-OSSS-CS and OSSS-CS change total scores (n = 216).