Emerging Brain-to-Content Technologies from Generative AI and Deep Representation Learning

Abstract

Rapid advances in generative artificial intelligence (AI) and deep representation learning have revolutionized numerous engineering applications in signal processing, computer vision, speech recognition and translation, and natural language processing due to amazingly powerful representation power (e.g., [1,2]). Generative AI-empowered tools, such as ChatGPT and Sora, have fundamentally changed the landscape of human-computer communications research. One emerging application along this line is to link the brain to the computer (i.e., brain-computer interface or BCI) and to develop paradigm-shift brain-to-content technologies. This BCI system upgrade (i.e., BCI 2.0) is empowered by generative AI and deep learning ("new engine") and large amounts of data ("gas"). In this article, we will revisit the old song sung in a new tune, highlight some state-of-the-art progresses, and briefly discuss the future outlook.

Keywords: brain-computer interface; deep representation learning; generative AI.

FIG 1.

Human-brain communications that decode contents from brain signals. In the brain module, neural signals (spike, MEG, EEG, ECoG, fMRI) can be acquired with various ranges of spatial and temporal resolution. In the encoding phase, the content module extracts features or embeddings of the data (e.g., speech, language, images) with assistance of pretrained deep neural networks. Neural embeddings and data embeddings are aligned so that a contextual mapping is established between two embedding spaces. In the decoding phase, the generation module produces the content using generative AI models and may adapt the generated content according to the preferred output style or domain using transfer learning.

FIG 2.

Illustration of a large landscape of generative AI. In the bottom, a taxonomy of generative models is shown according to their specific goals in density estimation. For instance, GAN generates synthetic data and compares the generated with real samples, reducing density estimation to a classification problem. Other generative models either directly estimate density (e.g., normalizing flow), or learn unnormalized density (e.g., EBM, score matching), or learn the approximation of density (e.g., VAE, DM). Some generative models use a mixed strategy for density estimation (e.g., autoregressive transformer, GPT, bi-LSTM, WaveNet). Among the explicit density models, there is a trade-off between generation quality and density modeling accuracy. Some explicit density models (such as DM and deep VAE) can also produce better generation quality than GAN.