Multi-modality artificial intelligence in digital pathology

Abstract

In common medical procedures, the time-consuming and expensive nature of obtaining test results plagues doctors and patients. Digital pathology research allows using computational technologies to manage data, presenting an opportunity to improve the efficiency of diagnosis and treatment. Artificial intelligence (AI) has a great advantage in the data analytics phase. Extensive research has shown that AI algorithms can produce more up-to-date and standardized conclusions for whole slide images. In conjunction with the development of high-throughput sequencing technologies, algorithms can integrate and analyze data from multiple modalities to explore the correspondence between morphological features and gene expression. This review investigates using the most popular image data, hematoxylin-eosin stained tissue slide images, to find a strategic solution for the imbalance of healthcare resources. The article focuses on the role that the development of deep learning technology has in assisting doctors' work and discusses the opportunities and challenges of AI.

Keywords: deep learning; digital pathology; multi-modality; whole slide images.

Figure 1

Workflow diagram for doctors, pathologists and computer scientists. Based on a doctor’s request for a pathological examination, the pathologist prepares sections of the patient’s tissue samples with stains, digitizes the sections and annotates them with WSIs information as ground truth. These images are combined with relevant electronic clinical information to serve as a training database for AI models that can be shared worldwide, relying on the Internet. Computer scientists train and test the models based on the ground truth and complete multiple performance testing experiments to evaluate model value. The entire research process can be explored in categories based on medical tasks, disease types, computer models and computer tasks. Mature models can provide assistance to the pathologists in clinical diagnosis and treatment [169].

Figure 2

Description of a unimodal generalized framework for AI application on digital pathology images. After the raw data collected at the hospital are annotated by pathologists, computer scientists complete the patches cut and enhance the data. The processed data are fed into a model pre-trained by ImageNet to achieve transfer learning of natural images to digital pathology images. The computer scientists train the model to fit the annotations and test the generalization of the model on an extra dataset that is also pre-processed. Finally, the model analysis results are used for statistical analysis as well as visual interpretation to explore their biological significance [169, 204].

Figure 3

Description of multimodal generalized ways for AI application on digital pathology images. (A) H&E images are predicted by feature extractor for gene expression, other stained images, or clinical information to achieve alignment of different kinds of information content. (B) H&E images and multiple types of data are jointly predicted by feature extractor for the target, and the kinds of feature dimensions need to be aligned in the feature extraction process [169].