Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2015 Feb 21:9416:941617.
doi: 10.1117/12.2081954. Epub 2015 Mar 17.

Developing a clinical utility framework to evaluate prediction models in radiogenomics

Yirong Wu  1 , Jie Liu  2 , Alejandro Munoz Del Rio  3 , David C Page  2 , Oguzhan Alagoz  4 , Peggy Peissig  5 , Adedayo A Onitilo  6 , Elizabeth S Burnside  1
Affiliations

Developing a clinical utility framework to evaluate prediction models in radiogenomics

Yirong Wu et al. Proc SPIE Int Soc Opt Eng. .

Abstract

Combining imaging and genetic information to predict disease presence and behavior is being codified into an emerging discipline called "radiogenomics." Optimal evaluation methodologies for radiogenomics techniques have not been established. We aim to develop a clinical decision framework based on utility analysis to assess prediction models for breast cancer. Our data comes from a retrospective case-control study, collecting Gail model risk factors, genetic variants (single nucleotide polymorphisms-SNPs), and mammographic features in Breast Imaging Reporting and Data System (BI-RADS) lexicon. We first constructed three logistic regression models built on different sets of predictive features: (1) Gail, (2) Gail+SNP, and (3) Gail+SNP+BI-RADS. Then, we generated ROC curves for three models. After we assigned utility values for each category of findings (true negative, false positive, false negative and true positive), we pursued optimal operating points on ROC curves to achieve maximum expected utility (MEU) of breast cancer diagnosis. We used McNemar's test to compare the predictive performance of the three models. We found that SNPs and BI-RADS features augmented the baseline Gail model in terms of the area under ROC curve (AUC) and MEU. SNPs improved sensitivity of the Gail model (0.276 vs. 0.147) and reduced specificity (0.855 vs. 0.912). When additional mammographic features were added, sensitivity increased to 0.457 and specificity to 0.872. SNPs and mammographic features played a significant role in breast cancer risk estimation (p-value < 0.001). Our decision framework comprising utility analysis and McNemar's test provides a novel framework to evaluate prediction models in the realm of radiogenomics.

Keywords: ROC methodology; breast imaging; expected utility; genetics; mammography.

PubMed Disclaimer

Figures

Figure 1
Figure 1
ROC curves for three models. Solid curve, Gail; dotted curve, Gail + SNP; dashed curve, Gail + SNP + BI-RADS
Figure 2
Figure 2
Expected utility curves of the three models. Solid curve, Gail; dotted curve, Gail + SNP; dashed curve, Gail + SNP + BI-RADS
Figure 3
Figure 3
Optimal operating points on ROC curves at maximum expected utility. Solid curve, Gail model; dotted curve, Gail + SNP model; dashed curve, Gail + SNP + BI-RADS model
See this image and copyright information in PMC

References

    1. Dai J, Hu Z, Jiang Y, et al. Breast cancer risk assessment with five independent genetic variants and two risk factors in Chinese women. Breast Cancer Res. 2012;14(1):R17. - PMC - PubMed
    1. Gail MH. Value of adding single-nucleotide polymorphism genotypes to a breast cancer risk model. J Natl Cancer Inst. 2009;101(13):959–963. - PMC - PubMed
    1. Lee C, Irwanto A, Salim A, et al. Breast cancer risk assessment using genetic variants and risk factors in a Singapore Chinese population. Breast Cancer Res. 2014;16(3):1–13. - PMC - PubMed
    1. Liu J, Page D, Nassif H, et al. Genetic variants improve breast cancer risk prediction on mammograms. American Medical Informatics Association Symposium (AMIA) 2013 - PMC - PubMed
    1. Liu J, Page D, Peissig P, et al. New genetic variants improve personalized breast cancer diagnosis. AMIA Summit on Translational Bioinformatics (AMIA-TBI) 2014 - PMC - PubMed