Triage of Patient Messages Sent to the Eye Clinic via the Electronic Medical Record: A Comparative Study on AI and Human Triage Performance

Abstract

Background/Objectives: Assess the ability of ChatGPT-4 (GPT-4) to effectively triage patient messages sent to the general eye clinic at our institution. Methods: Patient messages sent to the general eye clinic via MyChart were de-identified and then triaged by an ophthalmologist-in-training (MD) as well as GPT-4 with two main objectives. Both MD and GPT-4 were asked to direct patients to either general or specialty eye clinics, urgently or nonurgently, depending on the severity of the condition. Main Outcomes: GPT-4s ability to accurately direct patient messages to (1) a general or specialty eye clinic and (2) determine the time frame within which the patient needed to be seen (triage acuity). Accuracy was determined by comparing percent agreement with recommendations given by GPT-4 with those given by MD. Results: The study included 139 messages. Percent agreement between the ophthalmologist-in-training and GPT-4 was 64.7% for general/specialty clinic recommendation and 60.4% for triage acuity. Cohen's kappa was 0.33 and 0.67 for specialty clinic and triage urgency, respectively. GPT-4 recommended a triage acuity equal to or sooner than ophthalmologist-in-training for 93.5% of cases and recommended a less urgent triage acuity in 6.5% of cases. Conclusions: Our study indicates an AI system, such as GPT-4, should complement rather than replace physician judgment in triaging ophthalmic complaints. These systems may assist providers and reduce the workload of ophthalmologists and ophthalmic technicians as GPT-4 becomes more adept at triaging ophthalmic issues. Additionally, the integration of AI into ophthalmic triage could have therapeutic implications by ensuring timely and appropriate care, potentially improving patient outcomes by reducing delays in treatment. Combining GPT-4 with human expertise can improve service delivery speeds and patient outcomes while safeguarding against potential AI pitfalls.

Keywords: artificial intelligence; large language models; ophthalmology.

Figure 1

Scatterplot illustrating the comparison of triage recommendations between GPT-4 and an ophthalmologist-in-training for determining the appropriate specialty clinic for patient referrals. A small random offset was added to each value to avoid overlap of points.

Figure 2

Scatterplot illustrating the comparison of triage recommendations between GPT-4 and an ophthalmologist-in-training for determining the triage urgency in days. A small random offset was added to each value to avoid overlap of points.