Machine learning with multimodal data for COVID-19

Abstract

In response to the unprecedented global healthcare crisis of the COVID-19 pandemic, the scientific community has joined forces to tackle the challenges and prepare for future pandemics. Multiple modalities of data have been investigated to understand the nature of COVID-19. In this paper, MIDRC investigators present an overview of the state-of-the-art development of multimodal machine learning for COVID-19 and model assessment considerations for future studies. We begin with a discussion of the lessons learned from radiogenomic studies for cancer diagnosis. We then summarize the multi-modality COVID-19 data investigated in the literature including symptoms and other clinical data, laboratory tests, imaging, pathology, physiology, and other omics data. Publicly available multimodal COVID-19 data provided by MIDRC and other sources are summarized. After an overview of machine learning developments using multimodal data for COVID-19, we present our perspectives on the future development of multimodal machine learning models for COVID-19.

Keywords: COVID-19; Machine learning; Multimodal data.

Fig. 1

Multimodal data for COVID-19.

Fig. 2

Curation of the PETAL multimodal dataset involving two sources. The clinical data is hosted by Biodata Catalyst and access requires approval through dbGaP. The imaging data is hosted by MIDRC. Both data servers are powered by Gen3 and matching of the data from the two sources is achieved by a command line interface using the Gen3 Software Development Kit (SDK) on the Biomedical Research Hub, which requires Application Programming Interface (API) keys created on MIDRC and BioData catalyst.