Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Comparative Study
. 2015 Feb;24(1):107-40.
doi: 10.1177/0962280214537392. Epub 2014 Jun 11.

Statistical issues in the comparison of quantitative imaging biomarker algorithms using pulmonary nodule volume as an example

Nancy A Obuchowski  1 , Huiman X Barnhart  2 , Andrew J Buckler  3 , Gene Pennello  4 , Xiao-Feng Wang  5 , Jayashree Kalpathy-Cramer  6 , Hyun J Grace Kim  7 , Anthony P Reeves  8 ; Case Example Working Group
Affiliations
Comparative Study

Statistical issues in the comparison of quantitative imaging biomarker algorithms using pulmonary nodule volume as an example

Nancy A Obuchowski et al. Stat Methods Med Res. 2015 Feb.

Abstract

Quantitative imaging biomarkers are being used increasingly in medicine to diagnose and monitor patients' disease. The computer algorithms that measure quantitative imaging biomarkers have different technical performance characteristics. In this paper we illustrate the appropriate statistical methods for assessing and comparing the bias, precision, and agreement of computer algorithms. We use data from three studies of pulmonary nodules. The first study is a small phantom study used to illustrate metrics for assessing repeatability. The second study is a large phantom study allowing assessment of four algorithms' bias and reproducibility for measuring tumor volume and the change in tumor volume. The third study is a small clinical study of patients whose tumors were measured on two occasions. This study allows a direct assessment of six algorithms' performance for measuring tumor change. With these three examples we compare and contrast study designs and performance metrics, and we illustrate the advantages and limitations of various common statistical methods for quantitative imaging biomarker studies.

Keywords: agreement; bias; coverage probability; intraclass correlation coefficient; limits of agreement; repeatability; reproducibility.

PubMed Disclaimer

Figures

Figure 1
Figure 1
Plot of individual biases for 22 unique nodules ordered from smallest to largest volume (in mm3)
Figure 2
Figure 2. Plot of within-nodule SDs for 22 unique nodules ordered from smallest to largest volume (in mm3)
Note that there is no SD estimate for nodule #6 because there was only one measurement for this nodule.
Figure 3
Figure 3
Coverage probability curves by group and algorithm
Figure 4
Figure 4
Images of three moderate-sized nodules from the Volcano study: left□column, a central slice through the nodule; central column, a montage of□all the images slices that contain the nodule; right column, a 3D□visualization of an algorithm segmentation. First and second rows show□real nodules, the third row shows a spherical phantom nodule within an□anthropomorphic phantom.
Figure 5
Figure 5
Figure 6
Figure 6
Figure 7
Figure 7
Limits of agreement
Figure 8
Figure 8
Coverage Probability
See this image and copyright information in PMC

References

    1. Radiological Society of North America (RSNA) Quantitative Imaging Biomarkers Alliance (QIBA) 2010 Quantitative imaging and imaging biomarkers. http://www.rsna.org/research/qiba.cfm.
    1. Buckler AJ. Medical Imaging Continuum: Path Forward for Advancing the Uses of Medical Imaging in the Development of New Biopharmaceutical Products. DIA; Bethesda, MD: Oct 2-3, 2008. Quantitative Imaging Biomarker Alliance on volumetric CT (presentation)
    1. Buckler AJ, et al. A collaborative enterprise for multi-stakeholder participation in the advancement of quantitative imaging. Radiol. 2011;258(3):906–914. - PubMed
    1. QIBA Metrology Working Group The emerging science of quantitative imaging biomarkers: terminology and definitions for scientific studies and for regulatory submissions. Submitted to SMMR. - PubMed
    1. Performance Working Group Quantitative imaging biomarkers: a review of statistical methods for technical performance assessment. Submitted to SMMR. - PMC - PubMed

Publication types