Identifying Kidney Stone Risk Factors Through Patient Experiences With a Large Language Model: Text Analysis and Empirical Study

Abstract

Background: Kidney stones, a prevalent urinary disease, pose significant health risks. Factors like insufficient water intake or a high-protein diet increase an individual's susceptibility to the disease. Social media platforms can be a valuable avenue for users to share their experiences in managing these risk factors. Analyzing such patient-reported information can provide crucial insights into risk factors, potentially leading to improved quality of life for other patients.

Objective: This study aims to develop a model KSrisk-GPT, based on a large language model (LLM) to identify potential kidney stone risk factors from web-based user experiences.

Methods: This study collected data on the topic of kidney stones on Zhihu in the past 5 years and obtained 11,819 user comments. Experts organized the most common risk factors for kidney stones into six categories. Then, we use the least-to-most prompting in the chain-of-thought prompting to enable GPT-4.0 to think like an expert and ask GPT to identify risk factors from the comments. Metrics, including accuracy, precision, recall, and F₁-score, were used to evaluate the performance of such a model.

Results: Our proposed method outperforms other models in identifying comments containing risk factors with 95.9% accuracy and F₁-score, with a precision of 95.6% and a recall of 96.2%. Out of the 863 comments identified with risk factors, our analysis showed the most mentioned risk factors for kidney stones in Zhihu user discussions, mainly including dietary habits (high protein, high calcium intake), insufficient water intake, genetic factors, and lifestyle. In addition, new potential risk factors were discovered with GPT, such as excessive use of supplements like vitamin C and calcium, laxatives, and hyperparathyroidism.

Conclusions: Comments from social media users offer a new data source for disease prevention and understanding patient journeys. Our method not only sheds light on using LLMs to efficiently summarize risk factors from social media data but also on LLMs' potential to identify new potential factors from the patient's perspective.

Keywords: GPT-4; chronic disease management; health risk identification; large language model; text analysis.

Figure 1

A method for identifying kidney stone risk factors based on GPT-4. (A) Data collection and processing, aiming to obtain a clean dataset. (B) Professional doctors participate in the construction of LLMs, summarize the risk factors of kidney stones, and organize related topic words through the latest relevant literature and medical guidelines. (C) Identification of risk factors, using the LtM prompt engineering method and combining the above-mentioned Risk-factors and Topic-words to identify risk factors in user comments. LLM: large language model; LtM: least-to-most.