Real-World Evaluation of AI-Driven Diabetic Retinopathy Screening in Public Health Settings: Validation and Implementation Study

Abstract

Background: Artificial intelligence (AI) algorithms offer an effective solution to alleviate the burden of diabetic retinopathy (DR) screening in public health settings. However, there are challenges in translating diagnostic performance and its application when deployed in real-world conditions.

Objective: This study aimed to assess the technical feasibility of integration and diagnostic performance of validated DR screening (DRS) AI algorithms in real-world outpatient public health settings.

Methods: Prior to integrating an AI algorithm for DR screening, the study involved several steps: (1) Five AI companies, including four from India and one international company, were invited to evaluate their diagnostic performance using low-cost nonmydriatic fundus cameras in public health settings; (2) The AI algorithms were prospectively validated on fundus images from 250 people with diabetes mellitus, captured by a trained optometrist in public health settings in Chandigarh Tricity in North India. The performance evaluation used diagnostic metrics, including sensitivity, specificity, and accuracy, compared to human grader assessments; (3) The AI algorithm with better diagnostic performance was integrated into a low-cost screening camera deployed at a community health center (CHC) in the Moga district of Punjab, India. For AI algorithm analysis, a trained health system optometrist captured nonmydriatic images of 343 patients.

Results: Three web-based AI screening companies agreed to participate, while one declined and one chose to withdraw due to low specificity identified during the interim analysis. The three AI algorithms demonstrated variable diagnostic performance, with sensitivity (60%-80%) and specificity (14%-96%). Upon integration, the better-performing algorithm AI-3 (sensitivity: 68%, specificity: 96, and accuracy: 88·43%) demonstrated high sensitivity of image gradability (99.5%), DR detection (99.6%), and referral DR (79%) at the CHC.

Conclusions: This study highlights the importance of systematic AI validation for responsible clinical integration, demonstrating the potential of DRS to improve health care access in resource-limited public health settings.

Keywords: Screening, public health settings; artificial intelligence; diabetic retinopathy; screening; validation, implementation, integration.

Figure 1.. Study flow chart. Implementation phase occurred after the completion of the validation phase (ie, not simultaneously). AI: artificial intelligence; CHC: community health center; DR: diabetic retinopathy; PHC: primary health center; PPV: positive predictive value; NPV: negative predictive value; RDR: referable diabetic retinopathy.

Figure 2.. Image quality of artificial intelligence (AI) algorithms as compared to the reference standard (RS). DR: diabetic retinopathy.

Figure 3.. Diabetic retinopathy screening outputs of AI as compared to the reference standard at Community Health Center Moga. AI: artificial intelligence; DME: diabetic macular edeme; DR: diabetic retinopathy; PDR: proliferative diabetic retinopathy; NPDR: nonproliferative diabetic retinopathy.

Figure 4.. Agreement and kappa statistics for image quality and DR grade between AI and reference standard. AI: artificial intelligence; DR: diabetic retinopathy; RS: reference standard.