Protein language models trained on multiple sequence alignments learn phylogenetic relationships

Abstract

Self-supervised neural language models with attention have recently been applied to biological sequence data, advancing structure, function and mutational effect prediction. Some protein language models, including MSA Transformer and AlphaFold's EvoFormer, take multiple sequence alignments (MSAs) of evolutionarily related proteins as inputs. Simple combinations of MSA Transformer's row attentions have led to state-of-the-art unsupervised structural contact prediction. We demonstrate that similarly simple, and universal, combinations of MSA Transformer's column attentions strongly correlate with Hamming distances between sequences in MSAs. Therefore, MSA-based language models encode detailed phylogenetic relationships. We further show that these models can separate coevolutionary signals encoding functional and structural constraints from phylogenetic correlations reflecting historical contingency. To assess this, we generate synthetic MSAs, either without or with phylogeny, from Potts models trained on natural MSAs. We find that unsupervised contact prediction is substantially more resilient to phylogenetic noise when using MSA Transformer versus inferred Potts models.

Fig. 1. MSA Transformer: column attentions and Hamming distances.

a MSA Transformer is trained using the masked language modeling objective of filling in randomly masked residue positions in MSAs. For each residue position in an input MSA, it assigns attention scores to all residue positions in the same row (sequence) and column (site) in the MSA. These computations are performed by 12 independent row/column attention heads in each of 12 successive layers of the network. b Our approach for Hamming distance matrix prediction from the column attentions computed by the trained MSA Transformer model, using a natural MSA as input. For each i = 1, …, M, j = 0, …, L and l = 1, …, 12, the embedding vector $x_{ij}^{\left(l\right)}$ is the i-th row of the matrix $X_j^{\left(l\right)}$ defined in “Methods – MSA Transformer and column attention”, and the column attentions are computed according to Eqs. (2) and (3).

Fig. 2. Fitting logistic models to predict Hamming distances separately in each MSA.

The column-wise means of MSA Transformer’s column attention heads are used to predict normalised Hamming distances as probabilities in a logistic model. Each MSA is randomly split into a training set comprising 70% of its sequences and a test set composed of the remaining sequences. For each MSA, a logistic model is trained on all pairwise distances in the training set. Regression coefficients are shown for each layer and attention head (first column), as well as their absolute values averaged over heads for each layer (second column). For four example MSAs, ground truth Hamming distances are shown in the upper triangle (blue) and predicted Hamming distances in the lower triangle and diagonal (green), for the training and test sets (third and fourth columns). Darker shades correspond to larger Hamming distances.

Fig. 3. Pearson correlations between regression coefficients in larger MSAs.

Sufficiently deep (≥ 100 sequences) and long (≥ 30 residues) MSAs are considered (mean/min/max Pearson correlations: 0.80/0.69/0.87).

Fig. 4. Fitting a single logistic model to predict Hamming distances.

Our collection of 15 MSAs is split into a training set comprising 12 of them and a test set composed of the remaining 3. A logistic regression is trained on all pairwise distances within each MSA in the training set. Regression coefficients (first panel) and their absolute values averaged over heads for each layer (second panel) are shown as in Fig. 2. For the three test MSAs, ground truth Hamming distances are shown in the upper triangle (blue) and predicted Hamming distances in the lower triangle and diagonal (green), also as in Fig. 2 (last three panels). We further report the R² coefficients of determination for the regressions on these test MSAs—see also Supplementary Table 2.

Fig. 5. Correlations from coevolution and from phylogeny in MSAs.

a Natural selection on structure and function leads to correlations between residue positions in MSAs (coevolution). b Potts models, also known as DCA, aim to capture these correlations in their pairwise couplings. c Historical contingency can lead to correlations even in the absence of structural or functional constraints.