A Workflow for Meaningful Interpretation of Classification Results from Handheld Ambient Mass Spectrometry Analysis Probes

Abstract

While untargeted analysis of biological tissues with ambient mass spectrometry analysis probes has been widely reported in the literature, there are currently no guidelines to standardize the workflows for the experimental design, creation, and validation of molecular models that are utilized in these methods to perform class predictions. By drawing parallels with hurdles that are faced in the field of food fraud detection with untargeted mass spectrometry, we provide a stepwise workflow for the creation, refinement, evaluation, and assessment of the robustness of molecular models, aimed at meaningful interpretation of mass spectrometry-based tissue classification results. We propose strategies to obtain a sufficient number of samples for the creation of molecular models and discuss the potential overfitting of data, emphasizing both the need for model validation using an independent cohort of test samples, as well as the use of a fully characterized feature-based approach that verifies the biological relevance of the features that are used to avoid false discoveries. We additionally highlight the need to treat molecular models as "dynamic" and "living" entities and to further refine them as new knowledge concerning disease pathways and classifier feature noise becomes apparent in large(r) population studies. Where appropriate, we have provided a discussion of the challenges that we faced in our development of a 10 s cancer classification method using picosecond infrared laser mass spectrometry (PIRL-MS) to facilitate clinical decision-making at the bedside.

Keywords: ambient mass spectrometry; lipidomics; picosecond infrared laser mass spectrometry; rapid pathology.

Figure 1

Model validation using mixed class permutations. (A) PCA-LDA score plot of a model created from 10 independent mouse kidney (442, 10-s PIRL-MS sampling events), 10 independent mouse brain (446, 10-s PIRL-MS sampling events), and 10 independent mouse liver (409, 10-s PIRL-MS sampling events) samples. The data for this model were acquired as described previously using the reported hardware [46] and processed using 1/5th (20%) of the number of data points as the maximum number of PCA components (this was equal to 258 for 100% data usage). As shown here, a separate grouping of data points between classes is seen. (B) The PCA-LDA scores plot of a permuted model with mixed classes, wherein each group contains an equal representation of data from all other classes. This model does not show any separation of data, suggesting that the separation seen in the class data in panel (A) (with true class annotations) is significant. This model was processed using the same parameters that were used for panel (A). This is key. Panel (C) shows the PCA-LDA score plot of 409 data points (sampling events) from 10 independent mouse livers, analyzed in panels (A,B) and divided equally between three repetition classes, each containing 132, 138, and 139 events, respectively. This model serves as a control and shows a similar degree of mixing as what was shown for the permuted false class model of panel (A). Figure 2 shows the cross-validation statistics of these models (at 10% sample use intervals) from the 20% leave-out test and suggests 97.84%, 44%, and 51% for panel (A–C) models, respectively, at 100% data usage. For Figure (A,B) models, the maximum number of PCA components was 258. Reducing this to only 25 also resulted in cross-validation of 99.69% for (A), suggesting that when the inter-class variance is large and the intra-class variance is relatively small, way fewer than maximum (1/5th of the data) PCA component capture most of the data variance. This will be evident in the “Scree plots”. The concordance of panel (B,C)’s cross-validation statistics suggests that the permuted model bearing mixed annotations in each class effectively reports the cross-validation statistics of a dataset that is equivalent to that of multiple sampling events from the mouse liver, an organ that is highly molecularly homogeneous and bears little intra-class variability in its 10 s PIRL-MS signatures. Note that in this example, less than 50 specimens per class provided reliable performance due to large inter-class and small intra-class variance in the mass spectral data.

Figure 2

Learning curves created based on cross-validation statistics of Figure 1 PCA-LDA models. Cross-validation statistics of a series of PCA-LDA models created from various datasets with 10% increments of total data usage for Figure 1 models (where 1 denotes 10% data usage and 10 denotes 100%). These learning curves suggest that for models composed of highly molecularly distinct classes (such as Figure 1A), shown in panel (A), suitable cross-validation can be achieved at very low data utilization levels, past which the addition of further data fails to drastically impact the model’s predictive power. For permuted models that consistently sample the same heterogeneous class content across all data usage levels (Figure 1B), a poor(er) performance is seen, where in a similar vein, the addition of further data does not improve the performance (Panel (B)). To avoid overfitting, we adjusted the number of PCA components for each data usage accordingly. These values were 33 (for 10% data usage), 61, 89, 118, 151, 174, 198, 221, 239, and 258 (for 100% data usage) at each increment. In case the number of PCA components is not adjusted, for low data usage, overfitting takes place. Taking the learning curve of the Figure 1C model into consideration as a control for a highly homogeneous tissue-bearing low intra-class variability in panel (C), the learning curve from the permuted model of panel (B) appears to be similarly poorly performing. These learning curves were created from the same data that were utilized in Figure 1, as described previously [12]. Panel (D) shows the learning curve of Figure 1A, for which an inappropriately high number of PCA components of 258 across all data usages (except 10%, for which the maximum allowable was 165) was used. As can be seen here, overfitting takes place at low data usage following recovery and saturation at 80% of data usage.

Figure 3

An unusual lipid with ethylene glycol headgroup seen in PIRL-MS analysis of skin cancer. This previously unreported lipid was seen in our attempt to discover biomarker ions for skin cancer type differentiation with PIRL-MS [12]. In this figure, we are showing the MS/MS spectrum of the parent ion m/z 767.5232, obtained from extracts made from the plume of PIRL laser collected from cancer tissue by capture on a filter paper, as described in [12], and subsequently analyzed with ElectroSpray Ionization (ESI) with high-resolution mass spectrometry [12] post-chromatography. Surprisingly, this ion was not seen in lipid extracts made directly from the tissue, which may further imply a role for the desorption source in its formation (pending future additional clarifying experiments to be performed). This unusual headgroup is not reported in LipidMaps [97], which is routinely used by us and many other groups to verify tentative identity assignments for lipidic species.

Figure 4

The workflow for creation, evaluation, and refinement of molecular models for biological tissue type classification with untargeted mass spectrometry. Here, we have summarized our vision for how molecular models can best be created evaluated and refined towards rigorous use in future clinical decision-making (additional details are provided in the text regarding each step). While the provided pictorial summary largely captures the lessons learned throughout PIRL-MS developments in our group, the principles that are summarized in this figure may be relevant to other untargeted MS analysis approaches used in parallel or closely related studies.