Prospective Role of Foundation Models in Advancing Autonomous Vehicles

Abstract

With the development of artificial intelligence and breakthroughs in deep learning, large-scale foundation models (FMs), such as generative pre-trained transformer (GPT), Sora, etc., have achieved remarkable results in many fields including natural language processing and computer vision. The application of FMs in autonomous driving holds considerable promise. For example, they can contribute to enhancing scene understanding and reasoning. By pre-training on rich linguistic and visual data, FMs can understand and interpret various elements in a driving scene, and provide cognitive reasoning to give linguistic and action instructions for driving decisions and planning. Furthermore, FMs can augment data based on the understanding of driving scenarios to provide feasible scenes of those rare occurrences in the long tail distribution that are unlikely to be encountered during routine driving and data collection. The enhancement can subsequently lead to improvement in the accuracy and reliability of autonomous driving systems. Another testament to the potential of FMs' applications lies in world models, exemplified by the DREAMER series, which showcases the ability to comprehend physical laws and dynamics. Learning from massive data under the paradigm of self-supervised learning, world models can generate unseen yet plausible driving environments, facilitating the enhancement in the prediction of road users' behaviors and the off-line training of driving strategies. In this paper, we synthesize the applications and future trends of FMs in autonomous driving. By utilizing the powerful capabilities of FMs, we strive to tackle the potential issues stemming from the long-tail distribution in autonomous driving, consequently advancing overall safety in this domain.

Fig. 1.

Scaling laws [12].

Fig. 2.

Emergent abilities of LLMs [11]. (A) to (H) represent different downstream tasks. (A) 3-digit addition/subtraction, 2-digit multiplication. (B) Transliteration from the international phonetic alphabet. (C) Recovering scrambled words. (D) Persian question-answering. (E) Answering questions truthfully. (F) Mapping conceptual domains. (G) Massive multi-task language understanding. (H) Word in context semantic understanding. Each point is a separate LLM. The dotted line represents random performance.

Fig. 3.

The pipeline diagram for the supervised end-to-end autonomous driving system with a pretraining backbone. Multi-modal sensing information is input to the pretraining backbone to extract features, after which it enters into the framework of autonomous driving algorithms built by various methods to realize tasks, such as planning/control, to accomplish end-to-end autonomous driving tasks.

Fig. 4.

The pipeline diagram for enhancing autonomous driving leveraging FMs, where FMs refer to language models and vision models. FMs can learn perceptual information and utilize their powerful ability to understand the driving scenarios and reason to give language-guided instructions and driving actions to enhance autonomous driving.

Fig. 5.

For the application of LLMs to autonomous driving system decision-making, a typical pipeline is shown in this figure, referenced from DriveMLM [105].

Fig. 6.

For the application of LLMs to autonomous driving system planning, a typical pipeline is shown in this figure, referenced from LMDrive [119].

Fig. 7.

The pipeline diagram for enhancing autonomous driving with world models. The world models first learn the intrinsic evolutionary patterns by observing the traffic environment and then enhance autonomous driving by hooking up different decoders adapted to different driving tasks.

Fig. 8.

Comparison of the architecture of generative and non-generative methods [184]. (A) Generative architectures reconstruct a signal y from a compatible signal x using a decoder network conditioned on additional (possibly latent) variables z. (B) Joint-embedding predictive architectures predict the embeddings of a signal y from a compatible signal x using a predictor network conditioned on additional (possibly latent) variables z.