PEPhub: a database, web interface, and API for editing, sharing, and validating biological sample metadata

Abstract

Background: As biological data increase, we need additional infrastructure to share them and promote interoperability. While major effort has been put into sharing data, relatively less emphasis is placed on sharing metadata. Yet, sharing metadata is also important and in some ways has a wider scope than sharing data themselves.

Results: Here, we present PEPhub, an approach to improve sharing and interoperability of biological metadata. PEPhub provides an API, natural-language search, and user-friendly web-based sharing and editing of sample metadata tables. We used PEPhub to process more than 100,000 published biological research projects and index them with fast semantic natural-language search. PEPhub thus provides a fast and user-friendly way to finding existing biological research data or to share new data.

Availability: https://pephub.databio.org.

Keywords: metadata API; metadata machine learning; metadata sharing; metadata validation.

Figure 1:

PEPhub high-level architecture and project identification strategy. (A) PEPhub is backed by a Postgres database (left). It interfaces with the PEPhub server through a companion package called pepdbagent (middle). Web requests made by the web client or command-line interface are made via HTTP (right). (B) Workflow for automated GEO-to-PEPhub transfer using GEOfetch. We take advantage of scheduled GitHub Actions to automate new discovery of GEO accessions to upload. (C) PEPhub employs a {namespace}/{project}:{tag} nomenclature for sample table identification. Namespaces contain projects, which can be further distinguished with tags.

Figure 2:

Metadata sharing, discovery, and accessibility features. (A) PEPhub can convert metadata into JSON, csv, and txt output. (B) Using a pretrained sentence transformer, we periodically compute low-dimensional embeddings of all PEPs in PEPhub by mining text descriptions from the metadata. The resulting embeddings are then stored in Qdrant: a vector similarity engine and vector database. These embeddings are then compared against user-submitted queries. (C) Searching for a PEP in pephub using vector search happens in 5 steps. First, the user submits a natural-language query. Second, this query is embedded in real time on the server. Third, the resultant vector is used to query Qdrant for nearest neighbors. Fourth, Qdrant responds with the most similar vectors it has stored. Finally, the hits are returned to the client submitting the query.

Figure 3:

Metadata privacy and validation features. (A) Users have read access to all namespaces but write access only to their namespaces (left). Other users are not permitted to modify a PEP in any user namespace other than their own. PEPhub implements organizations through GitHub. Members of an organization are automatically granted write access to all PEPs that belong to that organization (right). (B) Validation on PEPhub is made easy with the integration of eido. PEPs in PEPub can be validated using the web-based validator UI, the metadata builder, or programmatic endpoints.

Figure 4:

Metadata management comparison chart. PEPhub compares favorably to alternative metadata management systems. ^aWhile open source, no clear documentation exists for self-hosting an instance.