Multimodal MALDI imaging mass spectrometry for improved diagnosis of melanoma

Abstract

Imaging mass spectrometry (IMS) provides promising avenues to augment histopathological investigation with rich spatio-molecular information. We have previously developed a classification model to differentiate melanoma from nevi lesions based on IMS protein data, a task that is challenging solely by histopathologic evaluation. Most IMS-focused studies collect microscopy in tandem with IMS data, but this microscopy data is generally omitted in downstream data analysis. Microscopy, nevertheless, forms the basis for traditional histopathology and thus contains invaluable morphological information. In this work, we developed a multimodal classification pipeline that uses deep learning, in the form of a pre-trained artificial neural network, to extract the meaningful morphological features from histopathological images, and combine it with the IMS data. To test whether this deep learning-based classification strategy can improve on our previous results in classification of melanocytic neoplasia, we utilized MALDI IMS data with collected serial H&E stained sections for 331 patients, and compared this multimodal classification pipeline to classifiers using either exclusively microscopy or IMS data. The multimodal pipeline achieved the best performance, with ROC-AUCs of 0.968 vs. 0.938 vs. 0.931 for the multimodal, unimodal microscopy and unimodal IMS pipelines respectively. Due to the use of a pre-trained network to perform the morphological feature extraction, this pipeline does not require any training on large amounts of microscopy data. As such, this framework can be readily applied to improve classification performance in other experimental settings where microscopy data is acquired in tandem with IMS experiments.

Fig 1. Overview of the histology-guided IMS sample preparation and processing workflow (a) and information about the final included number of samples and spots (b).

The workflow (a) leads into multimodal data analysis and classification, incorporating both chemical information from MALDI IMS and morphological information from H&E microscopy.

Fig 2

Data analysis workflow on an example tissue: top blue panel represents the unimodal IMS classification pipeline, where only IMS data are used for the downstream classification task; bottom orange panel shows the unimodal microscopy classification pipeline, where only morphology features are included for the melanoma diagnosis; In addition, intermediate embeddings from unimodal microscopy model were visualized via UMAP method: (a) 2-D embeddings of 16 patches (measured spots) from this example tissue with assigned colors based on pathologists’ annotations; (b) hyperspectral visualization of 3-D embeddings from all patches across the whole example tissue; middle fused panel is the multimodal strategy, where both IMS data and morphology features are used to distinguish melanoma from nevus #bins = 5558, P = 96, C = 512, VEM: vector embedding morphology model.

Fig 3. UMAP 2-D visualization of each input data across pipelines (unsupervised).

In the first row (a) preprocessed IMS data, (b) extracted morphological features from microscopy data and (c) an equally weighted combination of (a) and (b) are from the training dataset. The second row shows the UMAP visualization from the held-out test dataset. The assigned color of each spot is decided by its diagnosis from 3 dermatopathologists.

Fig 4. Classification results on training and held-out test sets.

All results are based on spot-level. (a) shows the classification performance of each pipeline on training and held-out test dataset. The results from the training set represent the mean value of the performance (with their standard deviation) after the nested-cross-validation process. (b)and(c) show the precision-recall and ROC cures plots of each pipeline on the held-out test set.

Fig 5. Misclassified cases by unimodal algorithms from the held-out test set.

Red and green text indicate model results that are incongruent and congruent with expert pathology assessment, respectively.