The role of metadata in reproducible computational research

Abstract

Reproducible computational research (RCR) is the keystone of the scientific method for in silico analyses, packaging the transformation of raw data to published results. In addition to its role in research integrity, improving the reproducibility of scientific studies can accelerate evaluation and reuse. This potential and wide support for the FAIR principles have motivated interest in metadata standards supporting reproducibility. Metadata provide context and provenance to raw data and methods and are essential to both discovery and validation. Despite this shared connection with scientific data, few studies have explicitly described how metadata enable reproducible computational research. This review employs a functional content analysis to identify metadata standards that support reproducibility across an analytic stack consisting of input data, tools, notebooks, pipelines, and publications. Our review provides background context, explores gaps, and discovers component trends of embeddedness and methodology weight from which we derive recommendations for future work.

Keywords: FAIR; RCR; containers; metadata; notebooks; ontologies; pipelines; provenance; replicability; reproducibility; reproducible computational research; reproducible research; semantic; software dependencies; workflows.

Figure 1

Whitaker’s matrix of reproducibility Whitaker's matrix of reproducibility; made available under the Creative Commons Attribution license (CC-BY 4.0).

Figure 2

Case studies in reproducible research The term "case studies" is used in a general sense to describe any study of reproducibility. A reproduction is an attempt to arrive at comparable results with identical data using computational methods described in a paper. A refactor involves refactoring existing code into frameworks and reproducible best practices while preserving the original data. A replication involves generating new data and applying existing methods to achieve comparable results. A test of robustness applies various protocols, workflows, statistical models, or parameters to a given dataset to study their effect on results. A census is a high-level tabulation conducted by a third party. A survey is a questionnaire sent to practitioners. A case narrative is an in-depth first-person account. An independent discussion uses a secondary independent author to interpret the results of a study as a means to improve inferential reproducibility.

Figure 3

Reproducibility Censuses like this one by Obels et al. measure data and code availability and reproducibility in this case over a corpus of 118 studies, 62 of which were psychology studies that had preregistered a Registered Report (RR).

Figure 4

Ecological metadata language Geographic and temporal EML metadata and the associated display on Knowledge Network for Biocomplexity (KNB) from Halpern et al.

Figure 5

STATO Concepts describing a linear mixed model used by STATO

Figure 6

Common workflow language Snippets of a COVID-19 variant detection CWL workflow and the workflow as viewed through the cwl-viewer. Note the EDAM file definitions.

Figure 7

Singapore Framework application profile model

Figure 8

tximeta The high-level schematic of tximeta.