GOProFormer: A Multi-Modal Transformer Method for Gene Ontology Protein Function Prediction

Abstract

Protein Language Models (PLMs) are shown to be capable of learning sequence representations useful for various prediction tasks, from subcellular localization, evolutionary relationships, family membership, and more. They have yet to be demonstrated useful for protein function prediction. In particular, the problem of automatic annotation of proteins under the Gene Ontology (GO) framework remains open. This paper makes two key contributions. It debuts a novel method that leverages the transformer architecture in two ways. A sequence transformer encodes protein sequences in a task-agnostic feature space. A graph transformer learns a representation of GO terms while respecting their hierarchical relationships. The learned sequence and GO terms representations are combined and utilized for multi-label classification, with the labels corresponding to GO terms. The method is shown superior over recent representative GO prediction methods. The second major contribution in this paper is a deep investigation of different ways of constructing training and testing datasets. The paper shows that existing approaches under- or over-estimate the generalization power of a model. A novel approach is proposed to address these issues, resulting in a new benchmark dataset to rigorously evaluate and compare methods and advance the state-of-the-art.

Keywords: gene ontology; multi-modal transformer; protein function.

Figure 1

The figure shows the proposed multi-modal Transformer model architecture. (a) depicts a black-box protein sequence modeling encoder. (b) shows the GO terms representation module using the Transformer encoder architecture.

Figure 2

The figures show the distribution of the number of labels per protein for the BP, CC and MF classes in the train-, val-, and test-set, respectively, considering the cutoff values of 150, 25, and 25. The test-set is generated considering the TDNK split, and the val-set is generated by randomly sampling 10% of the full dataset.

Figure 3

The train binary cross-entropy loss, validation binary cross-entropy loss and validation Fmax are shown for BP on each of the three dataset settings.

Figure 3

The train binary cross-entropy loss, validation binary cross-entropy loss and validation Fmax are shown for BP on each of the three dataset settings.

Figure 4

The train binary cross-entropy loss, validation binary cross-entropy loss and validation Fmax are shown for CC on each of the three dataset settings.

Figure 5

The train binary cross-entropy loss, validation binary cross-entropy loss and validation Fmax are shown for MF on each of the three dataset settings.