Transformer Architecture and Attention Mechanisms in Genome Data Analysis: A Comprehensive Review

Abstract

The emergence and rapid development of deep learning, specifically transformer-based architectures and attention mechanisms, have had transformative implications across several domains, including bioinformatics and genome data analysis. The analogous nature of genome sequences to language texts has enabled the application of techniques that have exhibited success in fields ranging from natural language processing to genomic data. This review provides a comprehensive analysis of the most recent advancements in the application of transformer architectures and attention mechanisms to genome and transcriptome data. The focus of this review is on the critical evaluation of these techniques, discussing their advantages and limitations in the context of genome data analysis. With the swift pace of development in deep learning methodologies, it becomes vital to continually assess and reflect on the current standing and future direction of the research. Therefore, this review aims to serve as a timely resource for both seasoned researchers and newcomers, offering a panoramic view of the recent advancements and elucidating the state-of-the-art applications in the field. Furthermore, this review paper serves to highlight potential areas of future investigation by critically evaluating studies from 2019 to 2023, thereby acting as a stepping-stone for further research endeavors.

Keywords: attention mechanism; bioinformatics; deep learning; genome data; genomics; natural language processing; sequence analysis; transcriptome data; transformer model.

Figure 1

Illustration of the transformer architecture and the attention mechanism. (A) Transformer structure; (B) Attention mechanism.

Figure 2

Distribution Patterns of Publication Years and Citation Frequencies (A) Distribution of Publication Years. (B) Distribution of Citation Frequencies. (C) Relationship between Citations and Publication Year.