Assessing GPT-4 for cell type annotation in single-cell RNA-seq analysis

Abstract

Here we demonstrate that the large language model GPT-4 can accurately annotate cell types using marker gene information in single-cell RNA sequencing analysis. When evaluated across hundreds of tissue and cell types, GPT-4 generates cell type annotations exhibiting strong concordance with manual annotations. This capability can considerably reduce the effort and expertise required for cell type annotation. Additionally, we have developed an R software package GPTCelltype for GPT-4's automated cell type annotation.

Fig. 1. Examples of GPT-4’s cell type annotation and comparisons with other methods.

a, Comparison of cell type annotations by human experts, GPT-4, and other automated methods. b, Example of GPT-4 annotating human prostate cells with increasing granularity. c, Example of GPT-4 annotating single, mixed and new cell types.

Fig. 2. Performance evaluation.

a, Average agreement scores for varying numbers of top differential genes, statistical tests for differential analysis, and prompt strategies. b, Proportion of cell types with varying agreement levels in each study and tissue, most abundant broad cell types, malignant cells, different cell population sizes, and major cell types versus cell subtypes. c, log₂-transformed ratio of type I (COL1A1 and COL1A2) and II (COL2A1) collagen gene expression. d,e, Comparison of average agreement scores (d) and running times (e). In e, n = 59 for GPT-4 and GPT-3.5 and n = 36 for ScType and SingleR. Each boxplot shows the distribution (center: median; bounds of box: first and third quartiles; bounds of whiskers: data points within 1.5× interquartile range from the box; minima; maxima) of running time. f, Financial cost of querying GPT-4 API versus cell type numbers. g, GPT-4’s performance in identifying mixed/single cell types and known/unknown cell types, and under different subsampling and noise levels in multiple simulation rounds (dots). h, Reproducibility of GPT-4 annotations. i, Consistency of agreement scores between two versions of GPT-4.