Machine learning and statistical inference in microbial population genomics

Abstract

The availability of large genome datasets has changed the microbiology research landscape. Analyzing such data requires computationally demanding analyses, and new approaches have come from different data analysis philosophies. Machine learning and statistical inference have overlapping knowledge discovery aims and approaches. However, machine learning focuses on optimizing prediction, whereas statistical inference focuses on understanding the processes relating variables. In this review, we outline the different aspirations, precepts, and resulting methodologies, with examples from microbial genomics. Emphasizing complementarity, we argue that the combination and synthesis of machine learning and statistics has potential for pathogen research in the big data era.

Fig. 1

Modelling and algorithmic approaches. Big data drawn from an example population that describes the objects from which data are randomly sampled (hand). This contains features, otherwise known as independent variables, predictors or regressors, and outcomes, otherwise known as dependent variables, labels, classes, or targets, whereby changes in the features lead to changes in the outcomes. Relating the two is the data generating process, or “nature”. Statistics (or, more precisely, data modelling in Breiman’s dichotomy [12]) aims to understand the underlying processes while ML (or, more precisely, algorithmic modelling in Breiman’s dichotomy) aims to faithfully reproduce the observed patterns to achieve optimal prediction, for instance

Fig. 2

Machine learning workflow in classification tasks. The data is split into training and testing, after which a suitable general-purpose algorithm is chosen, its hyper-parameters tuned and fitted to the training data. The performance of the fitted classifier is subsequently measured using a metric of choice