Development and Validation of an Artificial Intelligence System to Optimize Clinician Review of Patient Records

Abstract

Importance: Physicians are required to work with rapidly growing amounts of medical data. Approximately 62% of time per patient is devoted to reviewing electronic health records (EHRs), with clinical data review being the most time-consuming portion.

Objective: To determine whether an artificial intelligence (AI) system developed to organize and display new patient referral records would improve a clinician's ability to extract patient information compared with the current standard of care.

Design, setting, and participants: In this prognostic study, an AI system was created to organize patient records and improve data retrieval. To evaluate the system on time and accuracy, a nonblinded, prospective study was conducted at a single academic medical center. Recruitment emails were sent to all physicians in the gastroenterology division, and 12 clinicians agreed to participate. Each of the clinicians participating in the study received 2 referral records: 1 AI-optimized patient record and 1 standard (non-AI-optimized) patient record. For each record, clinicians were asked 22 questions requiring them to search the assigned record for clinically relevant information. Clinicians reviewed records from June 1 to August 30, 2020.

Main outcomes and measures: The time required to answer each question, along with accuracy, was measured for both records, with and without AI optimization. Participants were asked to assess overall satisfaction with the AI system, their preferred review method (AI-optimized vs standard), and other topics to assess clinical utility.

Results: Twelve gastroenterology physicians/fellows completed the study. Compared with standard (non-AI-optimized) patient record review, the AI system saved first-time physician users 18% of the time used to answer the clinical questions (10.5 [95% CI, 8.5-12.6] vs 12.8 [95% CI, 9.4-16.2] minutes; P = .02). There was no significant decrease in accuracy when physicians retrieved important patient information (83.7% [95% CI, 79.3%-88.2%] with the AI-optimized vs 86.0% [95% CI, 81.8%-90.2%] without the AI-optimized record; P = .81). Survey responses from physicians were generally positive across all questions. Eleven of 12 physicians (92%) preferred the AI-optimized record review to standard review. Despite a learning curve pointed out by respondents, 11 of 12 physicians believed that the technology would save them time to assess new patient records and were interested in using this technology in their clinic.

Conclusions and relevance: In this prognostic study, an AI system helped physicians extract relevant patient information in a shorter time while maintaining high accuracy. This finding is particularly germane to the ever-increasing amounts of medical data and increased stressors on clinicians. Increased user familiarity with the AI system, along with further enhancements in the system itself, hold promise to further improve physician data extraction from large quantities of patient health records.

Figure 1.. Data Flowchart and Artificial Intelligence (AI) System Pipeline

A, Forty-five readable patient referral records from gastroenterology clinicians were randomly obtained for the development of the system. They were separated into training, validation, and test sets. Three of the training set scans were manually annotated to identify information categories for the system to extract. Five records were used for physician evaluation: 4 for testing and 1 for collecting initial user feedback. B, The input is a scanned referral record, which is first partitioned into its constituent documents. The document is then classified into 1 of 9 categories (or undetermined): referral, note, laboratory, radiology, procedure, operative report, pathology, fax cover sheet, or insurance. The documents are ordered by their most recent date. Laboratory values are extracted and presented in a table, sorted by date. EHR indicates electronic health record.

Figure 2.. User Interface With Artificial Intelligence (AI)–Optimized Record

A, The original referral record PDF is displayed on the right of the interface. The AI output is shown on the left in 3 representative sections: B, the social information section, which contains smoking and allergy information; C, the radiology section, which displays items that are predicted to belong to the radiology category; and D, the laboratory table, which organizes by date the laboratory values extracted from the document. In practice, 1 section for each category would be predicted by the system to be in the document.

Figure 3.. Study Design and Record Assignment for Study

A, Clinicians were assigned 1 standard and 1 artificial intelligence (AI)–optimized record in random order. B, Combination of records assigned to each participant.

Figure 4.. Time Saved by Artificial Intelligence (AI) Optimization

A, Per-physician time taken for completion of questions for AI-optimized and standard review. Bars with an orange dot on the left and a blue dot on the right represent time saved with AI optimization; bars with a blue dot on the left and an orange dot on the right represent time lost. B, Time saved, adjusted for record size. The left side shows each physician’s time saved using AI-optimized review; the right side shows their time saved after using the time savings mixed model to standardize with a record size of 34 pages. Boxes represent quartiles (Q3-Q1), horizontal bars represent the median, and orange diamonds represent the mean. Error bars indicate the maximum and minimum values that are not outliers. Blue circles represent the time saved for each individual person. C, Association between standard review time and time saved with AI optimization. There was a correlation between the time physicians take to complete a standard review and the time saved with AI optimization (r = 0.80; P = .002).

Figure 5.. Subjective Feedback Results

After the 2 evaluation referral records, a subjective feedback survey was administered. Feedback from clinicians on software utility and performance on a 5-point Likert scale was generally positive. For any given survey question or statement, the absence of a percentage of respondents indicates that there were no responses for that category. AI indicates artificial intelligence.