Exploring and retrieving sequence and metadata for species across the tree of life with NCBI Datasets

Abstract

To explore complex biological questions, it is often necessary to access various data types from public data repositories. As the volume and complexity of biological sequence data grow, public repositories face significant challenges in ensuring that the data is easily discoverable and usable by the biological research community. To address these challenges, the National Center for Biotechnology Information (NCBI) has created NCBI Datasets. This resource provides straightforward, comprehensive, and scalable access to biological sequences, annotations, and metadata for a wide range of taxa. Following the FAIR (Findable, Accessible, Interoperable, and Reusable) data management principles, NCBI Datasets offers user-friendly web interfaces, command-line tools, and documented APIs, empowering researchers to access NCBI data seamlessly. The data is delivered as packages of sequences and metadata, thus facilitating improved data retrieval, sharing, and usability in research. Moreover, this data delivery method fosters effective data attribution and promotes its further reuse. This paper outlines the current scope of data accessible through NCBI Datasets and explains various options for exploring and downloading the data.

Fig. 1

NCBI Datasets interfaces for data access: NCBI Datasets offers web, command-line, and API access points that facilitate the search for and download of genomic sequences, annotations, and comprehensive metadata. These tools, including web-based and programmatic interfaces, guarantee consistent data retrieval from NCBI Datasets.

Fig. 2

Types of data packages: Three main categories of data packages are currently available: genome, gene, and virus. Users can customize the contents of any data package. Detailed information about each data package, including a list of available files and descriptions of each file, is available in NCBI Datasets documentation under Data packages (https://www.ncbi.nlm.nih.gov/datasets/docs/v2/reference-docs/data-packages/).

Fig. 3

Organism-focused data access: The NCBI Datasets taxonomy web page provides access to NCBI sequence and metadata data for that organism, including tabular views or assembled genome and annotated genes, download options and links to data in other NCBI databases.

Fig. 4

Organization of the datasets command-line tool: The datasets command-line tool can be used to browse and download NCBI Datasets data packages. There are two main subcommands: “download” for retrieving data packages and “summary” for displaying metadata. Each subcommand has multiple flags to help narrow data packages to the desired genomes or genes of interest. For an overview of datasets, dataformat, and installation instructions, see our Command-line tools documentation (https://www.ncbi.nlm.nih.gov/datasets/docs/v2/download-and-install/). (*) virus protein restricted to download of SARS-CoV-2 proteins.

Fig. 5

The datasets summary command: Diagram of the datasets command-line tool syntax for getting metadata for one or more genomes by taxon name.

Fig. 6

The datasets download command: Diagram of the datasets command-line tool syntax for downloading a genome data package for one or more genomes by taxon name.

Fig. 7

Organization of the assembly_data_report.jsonl data report. Select metadata can be retrieved from the data report using the dataformat command line tool. In this example, the organism name (red) and assembly accession (blue) can be extracted from the data report to generate a two-column tabular file.