Integrated image-based deep learning and language models for primary diabetes care

Abstract

Primary diabetes care and diabetic retinopathy (DR) screening persist as major public health challenges due to a shortage of trained primary care physicians (PCPs), particularly in low-resource settings. Here, to bridge the gaps, we developed an integrated image-language system (DeepDR-LLM), combining a large language model (LLM module) and image-based deep learning (DeepDR-Transformer), to provide individualized diabetes management recommendations to PCPs. In a retrospective evaluation, the LLM module demonstrated comparable performance to PCPs and endocrinology residents when tested in English and outperformed PCPs and had comparable performance to endocrinology residents in Chinese. For identifying referable DR, the average PCP's accuracy was 81.0% unassisted and 92.3% assisted by DeepDR-Transformer. Furthermore, we performed a single-center real-world prospective study, deploying DeepDR-LLM. We compared diabetes management adherence of patients under the unassisted PCP arm (n = 397) with those under the PCP+DeepDR-LLM arm (n = 372). Patients with newly diagnosed diabetes in the PCP+DeepDR-LLM arm showed better self-management behaviors throughout follow-up (P < 0.05). For patients with referral DR, those in the PCP+DeepDR-LLM arm were more likely to adhere to DR referrals (P < 0.01). Additionally, DeepDR-LLM deployment improved the quality and empathy level of management recommendations. Given its multifaceted performance, DeepDR-LLM holds promise as a digital solution for enhancing primary diabetes care and DR screening.

Fig. 1. Architecture of the DeepDR-LLM system.

The DeepDR-LLM system consists of two modules: (1) module I (LLM module), which provides individualized management recommendations for patients with diabetes; (2) module II (DeepDR-Transformer module), which performs image quality assessment, DR lesion segmentation and DR/DME grading from standard or portable fundus images. There are two modes of integrating module I and module II in the DeepDR-LLM system. In the physician-involved integration mode, the outputs of module II (that is, fundus image gradability; the lesion segmentation of microaneurysm, cotton-wool spot, hard exudate and hemorrhage; DR grade; and DME grade) could assist physicians in generating DR/DME diagnosis results (that is, fundus image gradability, DR grade, DME grade and the presence of lesions). In the automated integration mode, the DR/DME diagnosis results include fundus image gradability, DR grade, DME grade classified by module II, and the presence of lesions segmented out by module II. These DR/DME diagnosis results and other clinical metadata will be fed into module I to generate individualized management recommendations for people with diabetes.

Fig. 2. Study design overview for the DeepDR-LLM system evaluation.

a, Head-to-head comparative assessment of diabetes management recommendations generated by DeepDR-LLM, nontuned LLaMA, PCPs and endocrinology residents, using 100 cases randomly selected from CNDCS. b, Efficacy analysis of the DeepDR-Transformer module on multiethnic datasets of standard and portable fundus images. c, Utility evaluation of the DeepDR-Transformer module as an assistive tool for PCPs and professional graders in the detection of referable DR. d, Study design of a two-arm, real-world, prospective study to evaluate the impact of DeepDR-LLM on patients’ self-management behavior. In the outcome analysis, for substudy I, 253 participants in the unassisted PCP arm and 234 participants in the PCP+DeepDR-LLM arm were included; for substudy II, 154 participants in the unassisted PCP arm and 144 participants in the PCP+DeepDR-LLM arm were included.

Fig. 3. Head-to-head comparison between DeepDR-LLM, nontuned LLaMA, PCP and endocrinology resident in both English and Chinese.

a, Evaluators were invited to rate management recommendations for patients with diabetes, based on three domains, namely the extent of inappropriate content, the extent of missing content and the likelihood of possible harm, using 100 cases randomly selected from CNDCS. b, The total scores of management recommendations generated by LLaMA, DeepDR-LLM, PCPs and endocrinology residents, using 100 cases randomly selected from CNDCS. Box plot (n = 100), median and quartiles; whiskers, data range. The comparison was performed using two-sided Friedman tests. Post-hoc pairwise comparisons were performed using two-sided Wilcoxon signed-rank tests. P values for multiple comparisons were adjusted using the Bonferroni method. **P = 0.010, ***P < 0.001. Source data

Fig. 4. Receiver operating characteristic curves showing performance of DeepDR-Transformer alone versus PCPs (when unassisted and assisted by DeepDR-Transformer) in identifying referable DR.

a, Standard fundus images (500 eyes: 250 nonreferable eyes and 250 referable eyes) graded by PCPs in the urban area. b, Portable fundus images (500 eyes: 250 nonreferable eyes and 250 referable eyes) graded by PCPs in the urban area. c, Standard fundus images (500 eyes: 250 nonreferable eyes and 250 referable eyes) graded by PCPs in the rural area. d, Portable fundus images (500 eyes: 250 nonreferable eyes and 250 referable eyes) graded by PCPs in the rural area. Source data

Fig. 5. Quality and empathy ratings of the diabetes management recommendations by three consultant-level endocrinologists and 372 surveyed patients in the PCP+DeepDR-LLM arm.

a, Proportions of PCP, DeepDR-LLM and PCP+DeepDR-LLM’s recommendations being selected as the first-choice preference by consultant-level endocrinologists and patients (number of cases 372). Each of the three consultant-level endocrinologists was invited to evaluate all the 372 cases. The error bars show the Clopper–Pearson 95% CIs. b, Kernel density plots showing the quality and empathy ratings of PCP, DeepDR-LLM and PCP+DeepDR-LLM’s recommendations, as evaluated by three consultant-level endocrinologists (number of cases 372). Each of the three consultant-level endocrinologists was invited to evaluate all the 372 cases. c, Bar plots showing the quality and empathy ratings of PCP, DeepDR-LLM and PCP+DeepDR-LLM’s recommendations, as evaluated by the 372 surveyed patients (number of cases 372). Source data

Fig. 6. Envisioning the future of primary diabetes care with the clinical integration of the DeepDR-LLM system.

First, patients with diabetes undergo comprehensive evaluations that include medical history taking that can be augmented by automated voice-to-text technology, physical examinations, laboratory assessments and fundus imaging. Following this, the DeepDR-LLM system processes the accumulated clinical data to concurrently deliver DR screening results and tailored management recommendations for PCPs. Subsequently, augmented with these AI-derived insights, PCPs then offer treatment guidance and health education to patients, either in person or through teleconsultation services.

Extended Data Fig. 1. Schematic overview of the DeepDR-LLM system.

a, Model architecture of the DeepDR-Transformer module. b, Model architecture of the LLM module. DME, diabetic macular edema; DR, diabetic retinopathy; NPDR, non-proliferative diabetic retinopathy; PDR, proliferative diabetic retinopathy; LLM, large language model; LoRA, Low-Rank Adaptation; MLP, Multi-Layer Perceptron.

Extended Data Fig. 2. Study design of head-to-head comparison on diabetic management recommendations between large language models (DeepDR-LLM and LLaMA) and clinicians in both English and Chinese languages.

CNDCS, China National Diabetic Complications Study; PCP, primary care physician; ADA, American Diabetes Association; ICO, International Council of Ophthalmology.

Extended Data Fig. 3. Study design of the real-world, two-arm, prospective study.

For patients in the unassisted PCP arm, ten patients were diagnosed with both newly diagnosed diabetes and referable DR. For patients in the PCP+DeepDR-LLM arm, six patients were diagnosed with both newly diagnosed diabetes and referable DR. PCP, primary care physician; DR, diabetic retinopathy.

Extended Data Fig. 4. Study design of the post-deployment evaluation of management recommendations’ quality and level of empathy.

a, In the PCP + DeepDR-LLM arm, the DeepDR-LLM system was integrated into the clinical workflow. Initially, PCPs and DeepDR-LLM gave management recommendations independently. The recommendations given by DeepDR-LLM was automatically generated from electronic health systems, by extracting and analyzing the fundus images, medical history, physical examinations, and laboratory tests. Subsequently, PCPs edited their recommendations in text form by taking DeepDR-LLM’s recommendations into account. b, For participants in the PCP + DeepDR-LLM arm, they filled out a questionnaire investigating their opinions on three recommendations at the 4-week follow-up. Evaluators, including endocrinologists and surveyed participants, ranked these three recommendations and judged both ‘the quality of information provided’ (very poor, poor, acceptable, good, or very good) and ‘the empathy or bedside manner provided’ (not empathetic, slightly empathetic, moderately empathetic, empathetic, and very empathetic).