Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2014 Oct 1;56(10):1219-1232.
doi: 10.1016/j.infsof.2014.05.006.

Testing Scientific Software: A Systematic Literature Review

Upulee Kanewala  1 , James M Bieman  1
Affiliations

Testing Scientific Software: A Systematic Literature Review

Upulee Kanewala et al. Inf Softw Technol. .

Abstract

Context: Scientific software plays an important role in critical decision making, for example making weather predictions based on climate models, and computation of evidence for research publications. Recently, scientists have had to retract publications due to errors caused by software faults. Systematic testing can identify such faults in code.

Objective: This study aims to identify specific challenges, proposed solutions, and unsolved problems faced when testing scientific software.

Method: We conducted a systematic literature survey to identify and analyze relevant literature. We identified 62 studies that provided relevant information about testing scientific software.

Results: We found that challenges faced when testing scientific software fall into two main categories: (1) testing challenges that occur due to characteristics of scientific software such as oracle problems and (2) testing challenges that occur due to cultural differences between scientists and the software engineering community such as viewing the code and the model that it implements as inseparable entities. In addition, we identified methods to potentially overcome these challenges and their limitations. Finally we describe unsolved challenges and how software engineering researchers and practitioners can help to overcome them.

Conclusions: Scientific software presents special challenges for testing. Specifically, cultural differences between scientist developers and software engineers, along with the characteristics of the scientific software make testing more difficult. Existing techniques such as code clone detection can help to improve the testing process. Software engineers should consider special challenges posed by scientific software such as oracle problems when developing testing techniques.

Keywords: Scientific software; Software quality; Software testing; Systematic literature review.

PubMed Disclaimer

Figures

Figure 1
Figure 1
Percentage of studies that applied different testing methods
Figure 2
Figure 2
Number of testing methods applied by the studies
See this image and copyright information in PMC

References

    1. Abackerli AJ, Pereira PH, Calonego N., Jr A case study on testing CMM uncertainty simulation software (VCMM) Journal of the Brazilian Society of Mechanical Sciences and Engineering. 2010 Mar;32:8–14.
    1. Ackroyd K, Kinder S, Mant G, Miller M, Ramsdale C, Stephenson P. Scientific software development at a research facility. Software, IEEE. 2008 Jul-Aug;25(4):44–51.
    1. Afzal W, Torkar R, Feldt R. A systematic review of search-based testing for nonfunctional system properties. Information and Software Technology. 2009;51(6):957–976.
    1. Ammann P, Offutt J. Introduction to Software Testing, 1st Edition. New York, NY, USA: Cambridge University Press; 2008.
    1. Bagnara R, Carlier M, Gori R, Gotlieb A. Symbolic path-oriented test data generation for floating-point programs; Software Testing, Verification and Validation (ICST), 2013 IEEE Sixth International Conference on; 2013. pp. 1–10.

LinkOut - more resources