Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Feb 1;40(2):btae081.
doi: 10.1093/bioinformatics/btae081.

NFTest: automated testing of Nextflow pipelines

Yash Patel  1   2 , Chenghao Zhu  1   2   3 , Takafumi N Yamaguchi  1   2   3 , Yuan Zhe Bugh  1 , Mao Tian  1   2   3 , Aaron Holmes  1   2 , Sorel T Fitz-Gibbon  1   2   3 , Paul C Boutros  1   2   3   4   5
Affiliations

NFTest: automated testing of Nextflow pipelines

Yash Patel et al. Bioinformatics. .

Abstract

Motivation: The ongoing expansion in the volume of biomedical data has contributed to a growing complexity in the tools and technologies used in research with an increased reliance on complex workflows written in orchestration languages such as Nextflow to integrate algorithms into processing pipelines. The growing use of workflows involving various tools and algorithms has led to increased scrutiny of software development practices to avoid errors in individual tools and in the connections between them.

Results: To facilitate test-driven development of Nextflow pipelines, we created NFTest, a framework for automated pipeline testing and validation with customizability options for Nextflow features. It is open-source, easy to initialize and use, and customizable to allow for testing of complex workflows with test success configurable through a broad range of assertions. NFTest simplifies the testing burden on developers by automating tests once defined and providing a flexible interface for running tests to validate workflows. This reduces the barrier to rigorous biomedical workflow testing and paves the way toward reducing computational errors in biomedicine.

Availability and implementation: NFTest is an open-source Python framework under the GPLv2 license and is freely available at https://github.com/uclahs-cds/tool-NFTest. The call-sSNV Nextflow pipeline is available at: https://github.com/uclahs-cds/pipeline-call-sSNV.

PubMed Disclaimer

Conflict of interest statement

P.C.B. sits on the Scientific Advisory Boards of Intersect Diagnostics Inc., BioSymetrics Inc., and Sage Bionetworks. All other authors have no conflicts of interest to declare.

Figures

Figure 1.
Figure 1.
Architecture and workflow of testing. (a) Class diagram indicating relationships between environment variables, global settings, and case-specific settings. (b) Workflow of execution of test cases, indicating where the scripts are run and where assertions are made.
Figure 2.
Figure 2.
Workflow and testing of somatic SNV calling pipeline. (a) Somatic SNV calling pipeline flowchart through Mutect2, SomaticSniper, Strelka2, and MuSE algorithms culminating in intersection workflow. (b) Set of tests implemented for call-sSNV pipeline covering different inputs and algorithms. (c) Example intersection diagram of consensus variants; assertions are made to ensure the same diagram is produced through NFTest.
See this image and copyright information in PMC

References

    1. Baresi L, Pezzè M.. An introduction to software testing. Electron Notes Theor Comput Sci 2006;148:89–111.
    1. Broad Institute. Picard Toolkit. Broad Institute. GitHub repository. 2019. https://github.com/broadinstitute/picard.
    1. Chen H, Boutros PC.. VennDiagram: a package for the generation of highly-customizable Venn and Euler diagrams in R. BMC Bioinformatics 2011;12:35. - PMC - PubMed
    1. Cremin CJ, Dash S, Huang X. et al. Big data: historic advances and emerging trends in biomedical research. Curr Res Biotechnol 2022;4:138–51.
    1. Crusoe MR, Abeln S, Iosup A. et al. Methods included: standardizing computational reuse and portability with the common workflow language. Commun ACM 2022;65:54–63.

Publication types