Multi-Modal Fusion of Routine Care Electronic Health Records (EHR): A Scoping Review

Abstract

Background: Electronic health records (EHR) are now widely available in healthcare institutions to document the medical history of patients as they interact with healthcare services. In particular, routine care EHR data are collected for a large number of patients. These data span multiple heterogeneous elements (i.e., demographics, diagnosis, medications, clinical notes, vital signs, and laboratory results) which contain semantic, concept, and temporal information. Recent advances in generative learning techniques were able to leverage the fusion of multiple routine care EHR data elements to enhance clinical decision support.

Objective: A scoping review of the proposed techniques including fusion architectures, input data elements, and application areas is needed to synthesize variances and identify research gaps that can promote re-use of these techniques for new clinical outcomes.

Design: A comprehensive literature search was conducted using Google Scholar to identify high impact fusion architectures over multi-modal routine care EHR data during the period 2018 to 2023. The guidelines from the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) extension for scoping review were followed. The findings were derived from the selected studies using a thematic and comparative analysis.

Results: The scoping review revealed the lack of standard definition for EHR data elements as they are transformed into input modalities. These definitions ignore one or more key characteristics of the data including source, encoding scheme, and concept level. Moreover, in order to adapt to emergent generative learning techniques, the classification of fusion architectures should distinguish fusion from learning and take into consideration that learning can concurrently happen in all three layers of new fusion architectures (i.e., encoding, representation, and decision). These aspects constitute the first step towards a streamlined approach to the design of multi-modal fusion architectures for routine care EHR data. In addition, current pretrained encoding models are inconsistent in their handling of temporal and semantic information thereby hindering their re-use for different applications and clinical settings.

Conclusions: Current routine care EHR fusion architectures mostly follow a design-by-example methodology. Guidelines are needed for the design of efficient multi-modal models for a broad range of healthcare applications. In addition to promoting re-use, these guidelines need to outline best practices for combining multiple modalities while leveraging transfer learning and co-learning as well as semantic and temporal encoding.

Keywords: electronic health records; machine learning; modality; multi-modal fusion; transformers.

Figure 1.

Fusion architectures: (a) Encoding; (b) Decision; (c) Representation Learning.

Figure 2.

An encounter with representative routine care modalities, encodings, and concept levels.