Image fusion of mass spectrometry and microscopy: a multimodality paradigm for molecular tissue mapping

Abstract

We describe a predictive imaging modality created by 'fusing' two distinct technologies: imaging mass spectrometry (IMS) and microscopy. IMS-generated molecular maps, rich in chemical information but having coarse spatial resolution, are combined with optical microscopy maps, which have relatively low chemical specificity but high spatial information. The resulting images combine the advantages of both technologies, enabling prediction of a molecular distribution both at high spatial resolution and with high chemical specificity. Multivariate regression is used to model variables in one technology, using variables from the other technology. We demonstrate the potential of image fusion through several applications: (i) 'sharpening' of IMS images, which uses microscopy measurements to predict ion distributions at a spatial resolution that exceeds that of measured ion images by ten times or more; (ii) prediction of ion distributions in tissue areas that were not measured by IMS; and (iii) enrichment of biological signals and attenuation of instrumental artifacts, revealing insights not easily extracted from either microscopy or IMS individually.

Figure 1

Image fusion of imaging mass spectrometry (IMS) and microscopy. Image fusion generates a single image from two or more source images, combining the advantages of the different sensor types. The integration of IMS and optical microscopy is given as an example. The IMS-microscopy fusion image is a predictive modality that delivers both the chemical specificity of IMS and the spatial resolution of microscopy in one integrated whole. Each source image measures a different aspect of the content of a tissue sample. The fused image predicts the tissue content as if all aspects were observed concurrently.

Figure 2

Example of IMS-microscopy fusion. An ion image measured in mouse brain, describing the distribution of m/z 778.5 (identified as lipid PE(P-40:4)) at 100 μm spatial resolution (a), is integrated with an H&E-stained microscopy image measured from the same tissue sample at 10 μm resolution (b). By combining the information from both image types, the image fusion process can predict the ion distribution of m/z 778.5 at 10 μm resolution (c).

Figure 3

Prediction of the ion distribution of m/z 762.5 in mouse brain at 10 μm resolution from 100 μm IMS and 10 μm microscopy measurements (sharpening). This example in mouse brain fuses a measured ion image for m/z 762.5 (identified as lipid PE(16:0/22:6)) at 100 μm spatial resolution (a) with a measured H&E-stained microscopy image at 10 μm resolution (b), predicting the ion distribution of m/z 762.5 at 10 μm resolution (reconstr. score 82%) (c). For comparison, (d) shows a measured ion image for m/z 762.5 at 10 μm spatial resolution, acquired from a neighboring tissue section.

Figure 4

Prediction of the ion distributions of m/z 646.4 and 788.5 in mouse brain at 330 nm resolution from 10 μm IMS and 330 nm microscopy measurements (sharpening). Measured ion images acquired in mouse brain for m/z 646.4 and m/z 788.5 at 10 μm spatial resolution (a) are fused with an H&E-stained microscopy image measured at 0.33 μm resolution (b). The resulting IMS-microscopy model is combined with the microscopy measurements to predict the ion distributions of m/z 646.4 and m/z 788.5 at 330 nm resolution with an overall reconstruction score of respectively 75% and 76% (c).

Figure 5

Prediction of m/z 10,516 distribution in mouse brain areas not measured by IMS (out-of-sample prediction). An IMS-microscopy model is built on a tissue sub-area for which IMS is available at 100 μm resolution (a) and H&E-stained microscopy is available at 5 μm resolution (b). The model is then used to predict the distribution of m/z 10,516 in areas where no IMS was acquired and only microscopy is available (reconstr. score 88%) (d). (Non-sharpened version available in Supplementary Fig. 21)

Figure 6

Discovery of tissue features through multi-modal enrichment. An ion image for m/z 3,345 measured by IMS at 100 μm resolution in a rat kidney section (a) is fused with H&E stained microscopy acquired at 5 μm resolution (b) to produce an ion distribution prediction at 5 μm resolution (reconstr. score 85%) (c). Annotations a–c demonstrate multi-modal enrichment. If only IMS is considered, these features could be mistaken for ‘matrix hotspot’ noise. However, their successful propagation through the fusion process and their presence in the final fused image confirms they are genuine tissue features that are corroborated by another technology (in this case microscopy). If only microscopy is considered, these features are so faint that they would probably not be detected. Instead, they are only discovered by fusion with another data source. Annotation d demonstrates multi-modal attenuation. The lack of cross-modal support for this localized drop in ion intensity reduces confidence in the biological nature of this feature.