. 2020 May;295(2):328-338.

doi: 10.1148/radiol.2020191145. Epub 2020 Mar 10.

The Image Biomarker Standardization Initiative: Standardized Quantitative Radiomics for High-Throughput Image-based Phenotyping

Alex Zwanenburg^#¹, Martin Vallières^#¹, Mahmoud A Abdalah¹, Hugo J W L Aerts¹, Vincent Andrearczyk¹, Aditya Apte¹, Saeed Ashrafinia¹, Spyridon Bakas¹, Roelof J Beukinga¹, Ronald Boellaard¹, Marta Bogowicz¹, Luca Boldrini¹, Irène Buvat¹, Gary J R Cook¹, Christos Davatzikos¹, Adrien Depeursinge¹, Marie-Charlotte Desseroit¹, Nicola Dinapoli¹, Cuong Viet Dinh¹, Sebastian Echegaray¹, Issam El Naqa¹, Andriy Y Fedorov¹, Roberto Gatta¹, Robert J Gillies¹, Vicky Goh¹, Michael Götz¹, Matthias Guckenberger¹, Sung Min Ha¹, Mathieu Hatt¹, Fabian Isensee¹, Philippe Lambin¹, Stefan Leger¹, Ralph T H Leijenaar¹, Jacopo Lenkowicz¹, Fiona Lippert¹, Are Losnegård¹, Klaus H Maier-Hein¹, Olivier Morin¹, Henning Müller¹, Sandy Napel¹, Christophe Nioche¹, Fanny Orlhac¹, Sarthak Pati¹, Elisabeth A G Pfaehler¹, Arman Rahmim¹, Arvind U K Rao¹, Jonas Scherer¹, Muhammad Musib Siddique¹, Nanna M Sijtsema¹, Jairo Socarras Fernandez¹, Emiliano Spezi¹, Roel J H M Steenbakkers¹, Stephanie Tanadini-Lang¹, Daniela Thorwarth¹, Esther G C Troost¹, Taman Upadhaya¹, Vincenzo Valentini¹, Lisanne V van Dijk¹, Joost van Griethuysen¹, Floris H P van Velden¹, Philip Whybra¹, Christian Richter¹, Steffen Löck¹

Affiliations

Affiliation

¹ From OncoRay-National Center for Radiation Research in Oncology, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Helmholtz-Zentrum Dresden-Rossendorf, Fetscherstr 74, PF 41, 01307 Dresden, Germany (A.Z., S. Leger, E.G.C.T., C.R., S. Löck); National Center for Tumor Diseases (NCT), Partner Site Dresden, Germany: German Cancer Research Center (DKFZ), Heidelberg, Germany (A.Z.); Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden and Helmholtz Association/Helmholtz-Zentrum Dresden-Rossendorf (HZDR), Dresden, Germany (A.Z., S. Leger, E.G.C.T.); German Cancer Consortium (DKTK), Partner Site Dresden, and German Cancer Research Center (DKFZ), Heidelberg, Germany (A.Z., S. Leger, E.G.C.T., C.R., S. Löck); Medical Physics Unit, McGill University, Montréal, Canada (M.V., I.E.N.); Image Response Assessment Team Core Facility, Moffitt Cancer Center, Tampa, Fla (M.A.A.); Dana-Farber Cancer Institute, Brigham and Women's Hospital, and Harvard Medical School, Harvard University, Boston, Mass (H.J.W.L.A.); Institute of Information Systems, University of Applied Sciences Western Switzerland (HES-SO), Sierre, Switzerland (V.A., A.D., H.M.); Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, NY (A.A.); Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, Md (S.A.); Department of Radiology and Radiological Science, Johns Hopkins University, Baltimore, Md (S.A., A.R.); Center for Biomedical image Computing and Analytics (CBICA), University of Pennsylvania, Philadelphia, Pa (S.B., C.D., S.M.H., S.P.); Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pa (S.B., C.D., S.M.H., S.P.); Department of Pathology and Laboratory Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pa (S.B.); Department of Nuclear Medicine and Molecular Imaging, University of Groningen, University Medical Center Groningen (UMCG), Groningen, the Netherlands (R.J.B., R.B., E.A.G.P.); Radiology and Nuclear Medicine, VU University Medical Centre (VUMC), Amsterdam, the Netherlands (R.B.); Department of Radiation Oncology, University Hospital Zurich, University of Zurich, Zurich, Switzerland (M.B., M.Guckenberger, S.T.L.); Fondazione Policlinico Universitario "A. Gemelli" IRCCS, Rome, Italy (L.B., N.D., R.G., J.L., V.V.); Laboratoire d'Imagerie Translationnelle en Oncologie, Université Paris Saclay, Inserm, Institut Curie, Orsay, France (I.B., C.N., F.O.); Cancer Imaging Dept, School of Biomedical Engineering and Imaging Sciences, King's College London, London, United Kingdom (G.J.R.C., V.G., M.M.S.); Department of Nuclear Medicine and Molecular Imaging, Lausanne University Hospital, Lausanne, Switzerland (A.D.); Laboratory of Medical Information Processing (LaTIM)-team ACTION (image-guided therapeutic action in oncology), INSERM, UMR 1101, IBSAM, UBO, UBL, Brest, France (M.C.D., M.H., T.U.); Department of Radiation Oncology, the Netherlands Cancer Institute (NKI), Amsterdam, the Netherlands (C.V.D.); Department of Radiology, Stanford University School of Medicine, Stanford, Calif (S.E., S.N.); Department of Radiation Oncology, Physics Division, University of Michigan, Ann Arbor, Mich (I.E.N., A.U.K.R.); Surgical Planning Laboratory, Brigham and Women's Hospital and Harvard Medical School, Harvard University, Boston, Mass (A.Y.F.); Department of Cancer Imaging and Metabolism, Moffitt Cancer Center, Tampa, Fla (R.J.G.); Department of Medical Image Computing, German Cancer Research Center (DKFZ), Heidelberg, Germany (M. Götz, F.I., K.H.M.H., J.S.); The D-Lab, Department of Precision Medicine, GROW-School for Oncology and Developmental Biology, Maastricht University Medical Centre+, Maastricht, the Netherlands (P.L., R.T.H.L.); Section for Biomedical Physics, Department of Radiation Oncology, University of Tübingen, Germany (F.L., J.S.F., D.T.); Department of Clinical Medicine, University of Bergen, Bergen, Norway (A.L.); Department of Radiation Oncology, University of California, San Francisco, Calif (O.M.); University of Geneva, Geneva, Switzerland (H.M.); Department of Electrical Engineering, Stanford University, Stanford, Calif (S.N.); Department of Medicine (Biomedical Informatics Research), Stanford University School of Medicine, Stanford, Calif (S.N.); Departments of Radiology and Physics, University of British Columbia, Vancouver, Canada (A.R.); Department of Computational Medicine and Bioinformatics, University of Michigan, Ann Arbor, Mich (A.U.K.R.); Department of Radiation Oncology, University of Groningen, University Medical Center Groningen (UMCG), Groningen, the Netherlands (N.M.S., R.J.H.M.S., L.V.v.D.); School of Engineering, Cardiff University, Cardiff, United Kingdom (E.S., P.W.); Department of Medical Physics, Velindre Cancer Centre, Cardiff, United Kingdom (E.S.); Department of Radiotherapy and Radiation Oncology, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany (E.G.C.T., C.R., S. Löck), Helmholtz-Zentrum Dresden-Rossendorf, Institute of Radiooncology-OncoRay, Dresden, Germany (E.G.C.T., C.R.); Department of Nuclear Medicine, CHU Milétrie, Poitiers, France (T.U.); Department of Radiology, the Netherlands Cancer Institute (NKI), Amsterdam, the Netherlands (J.v.G.); GROW-School for Oncology and Developmental Biology, Maastricht University Medical Center, Maastricht, the Netherlands (J.v.G.); Department of Radiation Oncology, Dana-Farber Cancer Institute, Brigham and Women's Hospital, Harvard Medical School, Boston, Mass (J.v.G.); and Department of Radiology, Leiden University Medical Center (LUMC), Leiden, the Netherlands (F.H.P.v.V.).

^# Contributed equally.

PMID: 32154773
PMCID: PMC7193906
DOI: 10.1148/radiol.2020191145

The Image Biomarker Standardization Initiative: Standardized Quantitative Radiomics for High-Throughput Image-based Phenotyping

Alex Zwanenburg et al. Radiology. 2020 May.

. 2020 May;295(2):328-338.

doi: 10.1148/radiol.2020191145. Epub 2020 Mar 10.

Authors

Affiliation

¹ From OncoRay-National Center for Radiation Research in Oncology, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Helmholtz-Zentrum Dresden-Rossendorf, Fetscherstr 74, PF 41, 01307 Dresden, Germany (A.Z., S. Leger, E.G.C.T., C.R., S. Löck); National Center for Tumor Diseases (NCT), Partner Site Dresden, Germany: German Cancer Research Center (DKFZ), Heidelberg, Germany (A.Z.); Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden and Helmholtz Association/Helmholtz-Zentrum Dresden-Rossendorf (HZDR), Dresden, Germany (A.Z., S. Leger, E.G.C.T.); German Cancer Consortium (DKTK), Partner Site Dresden, and German Cancer Research Center (DKFZ), Heidelberg, Germany (A.Z., S. Leger, E.G.C.T., C.R., S. Löck); Medical Physics Unit, McGill University, Montréal, Canada (M.V., I.E.N.); Image Response Assessment Team Core Facility, Moffitt Cancer Center, Tampa, Fla (M.A.A.); Dana-Farber Cancer Institute, Brigham and Women's Hospital, and Harvard Medical School, Harvard University, Boston, Mass (H.J.W.L.A.); Institute of Information Systems, University of Applied Sciences Western Switzerland (HES-SO), Sierre, Switzerland (V.A., A.D., H.M.); Department of Medical Physics, Memorial Sloan Kettering Cancer Center, New York, NY (A.A.); Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, Md (S.A.); Department of Radiology and Radiological Science, Johns Hopkins University, Baltimore, Md (S.A., A.R.); Center for Biomedical image Computing and Analytics (CBICA), University of Pennsylvania, Philadelphia, Pa (S.B., C.D., S.M.H., S.P.); Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pa (S.B., C.D., S.M.H., S.P.); Department of Pathology and Laboratory Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pa (S.B.); Department of Nuclear Medicine and Molecular Imaging, University of Groningen, University Medical Center Groningen (UMCG), Groningen, the Netherlands (R.J.B., R.B., E.A.G.P.); Radiology and Nuclear Medicine, VU University Medical Centre (VUMC), Amsterdam, the Netherlands (R.B.); Department of Radiation Oncology, University Hospital Zurich, University of Zurich, Zurich, Switzerland (M.B., M.Guckenberger, S.T.L.); Fondazione Policlinico Universitario "A. Gemelli" IRCCS, Rome, Italy (L.B., N.D., R.G., J.L., V.V.); Laboratoire d'Imagerie Translationnelle en Oncologie, Université Paris Saclay, Inserm, Institut Curie, Orsay, France (I.B., C.N., F.O.); Cancer Imaging Dept, School of Biomedical Engineering and Imaging Sciences, King's College London, London, United Kingdom (G.J.R.C., V.G., M.M.S.); Department of Nuclear Medicine and Molecular Imaging, Lausanne University Hospital, Lausanne, Switzerland (A.D.); Laboratory of Medical Information Processing (LaTIM)-team ACTION (image-guided therapeutic action in oncology), INSERM, UMR 1101, IBSAM, UBO, UBL, Brest, France (M.C.D., M.H., T.U.); Department of Radiation Oncology, the Netherlands Cancer Institute (NKI), Amsterdam, the Netherlands (C.V.D.); Department of Radiology, Stanford University School of Medicine, Stanford, Calif (S.E., S.N.); Department of Radiation Oncology, Physics Division, University of Michigan, Ann Arbor, Mich (I.E.N., A.U.K.R.); Surgical Planning Laboratory, Brigham and Women's Hospital and Harvard Medical School, Harvard University, Boston, Mass (A.Y.F.); Department of Cancer Imaging and Metabolism, Moffitt Cancer Center, Tampa, Fla (R.J.G.); Department of Medical Image Computing, German Cancer Research Center (DKFZ), Heidelberg, Germany (M. Götz, F.I., K.H.M.H., J.S.); The D-Lab, Department of Precision Medicine, GROW-School for Oncology and Developmental Biology, Maastricht University Medical Centre+, Maastricht, the Netherlands (P.L., R.T.H.L.); Section for Biomedical Physics, Department of Radiation Oncology, University of Tübingen, Germany (F.L., J.S.F., D.T.); Department of Clinical Medicine, University of Bergen, Bergen, Norway (A.L.); Department of Radiation Oncology, University of California, San Francisco, Calif (O.M.); University of Geneva, Geneva, Switzerland (H.M.); Department of Electrical Engineering, Stanford University, Stanford, Calif (S.N.); Department of Medicine (Biomedical Informatics Research), Stanford University School of Medicine, Stanford, Calif (S.N.); Departments of Radiology and Physics, University of British Columbia, Vancouver, Canada (A.R.); Department of Computational Medicine and Bioinformatics, University of Michigan, Ann Arbor, Mich (A.U.K.R.); Department of Radiation Oncology, University of Groningen, University Medical Center Groningen (UMCG), Groningen, the Netherlands (N.M.S., R.J.H.M.S., L.V.v.D.); School of Engineering, Cardiff University, Cardiff, United Kingdom (E.S., P.W.); Department of Medical Physics, Velindre Cancer Centre, Cardiff, United Kingdom (E.S.); Department of Radiotherapy and Radiation Oncology, Faculty of Medicine and University Hospital Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany (E.G.C.T., C.R., S. Löck), Helmholtz-Zentrum Dresden-Rossendorf, Institute of Radiooncology-OncoRay, Dresden, Germany (E.G.C.T., C.R.); Department of Nuclear Medicine, CHU Milétrie, Poitiers, France (T.U.); Department of Radiology, the Netherlands Cancer Institute (NKI), Amsterdam, the Netherlands (J.v.G.); GROW-School for Oncology and Developmental Biology, Maastricht University Medical Center, Maastricht, the Netherlands (J.v.G.); Department of Radiation Oncology, Dana-Farber Cancer Institute, Brigham and Women's Hospital, Harvard Medical School, Boston, Mass (J.v.G.); and Department of Radiology, Leiden University Medical Center (LUMC), Leiden, the Netherlands (F.H.P.v.V.).

^# Contributed equally.

PMID: 32154773
PMCID: PMC7193906
DOI: 10.1148/radiol.2020191145

Abstract

Background Radiomic features may quantify characteristics present in medical imaging. However, the lack of standardized definitions and validated reference values have hampered clinical use. Purpose To standardize a set of 174 radiomic features. Materials and Methods Radiomic features were assessed in three phases. In phase I, 487 features were derived from the basic set of 174 features. Twenty-five research teams with unique radiomics software implementations computed feature values directly from a digital phantom, without any additional image processing. In phase II, 15 teams computed values for 1347 derived features using a CT image of a patient with lung cancer and predefined image processing configurations. In both phases, consensus among the teams on the validity of tentative reference values was measured through the frequency of the modal value and classified as follows: less than three matches, weak; three to five matches, moderate; six to nine matches, strong; 10 or more matches, very strong. In the final phase (phase III), a public data set of multimodality images (CT, fluorine 18 fluorodeoxyglucose PET, and T1-weighted MRI) from 51 patients with soft-tissue sarcoma was used to prospectively assess reproducibility of standardized features. Results Consensus on reference values was initially weak for 232 of 302 features (76.8%) at phase I and 703 of 1075 features (65.4%) at phase II. At the final iteration, weak consensus remained for only two of 487 features (0.4%) at phase I and 19 of 1347 features (1.4%) at phase II. Strong or better consensus was achieved for 463 of 487 features (95.1%) at phase I and 1220 of 1347 features (90.6%) at phase II. Overall, 169 of 174 features were standardized in the first two phases. In the final validation phase (phase III), most of the 169 standardized features could be excellently reproduced (166 with CT; 164 with PET; and 164 with MRI). Conclusion A set of 169 radiomics features was standardized, which enabled verification and calibration of different radiomics software. © RSNA, 2020 Online supplemental material is available for this article. See also the editorial by Kuhl and Truhn in this issue.

PubMed Disclaimer

Figures

**Figure 1:**
Flowchart of study overview. The workflow in a typical radiomics analysis starts with acquisition and reconstruction of a medical image. Subsequently, the image is segmented to define regions of interest (ROI). Afterward, radiomics software is used to process the image and to compute features that characterize an ROI. We focused on standardizing the image processing and feature computation steps. Standardization was performed within two iterative phases. In phase I, we used a specially designed digital phantom to obtain reference values for radiomics features directly. In phase II, a publicly available CT image in a patient with lung cancer was used to obtain reference values for features under predefined configurations of a standardized general radiomics image processing scheme. Standardization of image processing and feature computation steps in radiomics software was prospectively validated during phase III by assessing reproducibility of standardized features in a publicly available multimodality patient cohort of 51 patients with soft-tissue sarcoma. 18F-FDG = fluorine 18 fluorodeoxyglucose, T1w = T1-weighted.

**Figure 2:**
Flowchart of the general radiomics image processing scheme for computing radiomics features. Image processing starts with reconstructed images. These images are processed through several optional steps: data conversion (eg, conversion to standardized uptake values), image postacquisition processing (eg, image denoising), and image interpolation. Either the region of interest (ROI) is created automatically during the segmentation step, or an existing ROI is retrieved. The ROI is then interpolated as well, and intensity and morphologic masks are created as copies. The intensity mask may be resegmented according to intensity values to improve comparability of intensity ranges across a cohort. Radiomics features are then computed from the image masked by the ROI and its immediate neighborhood (local intensity features) or the ROI itself (all others). Image intensities are moreover discretized prior to computation of features from the intensity histogram (IH), intensity-volume histogram (IVH), gray-level co-occurrence matrix (GLCM), gray-level run-length matrix (GLRLM), gray-level size-zone matrix (GLSZM), gray-level distance-zone matrix (GLDZM), neighborhood gray-tone difference matrix (NGTDM), and neighboring gray-level dependence matrix (NGLDM) families. All processing steps from image interpolation to the computation of radiomics features were evaluated in this study.

**Figure 3:**
Bar graphs depict participation and radiomics feature coverage by research teams. A, Graph shows the number of research teams at each analysis time point during the two phases of the iterative standardization process. Teams computed features without prior image processing (phase I) and after image processing (phase II), with the aim of finding reference values for a feature. Consensus on the validity of reference values was assessed at each time point, the time between which was variable (arbitrary unit [arb. unit]). B, Graph shows the final coverage of radiomics features implemented by each team in phase I, as well as the team’s ability to reproduce the reference value of a feature. We were unable to obtain reliable reference values for five features (no ref. value). The teams are listed in Appendix E1 (online). BCOM = Institute of Research and Technology b<>com, Brest, CaPTk = Cancer Imaging Phenomics Toolkit, CERR = Computational Environment for Radiological Research, KCL = King’s College London, LUMC = Leiden University Medical Center, MAASTRO = Maastro, Maastricht, the Netherlands, MaCha = Marie-Charlotte Desseroit, MIRP = Medical Image Radiomics Processor, MITK = Medical Imaging Interaction Toolkit, QIFE = Quantitative Image Feature Engine, RaCaT = Radiomics Calculator, SERA = Standardized Environment for Radiomics Analysis, UCSF = University of California, San Francisco, UMCG = University Medical Center Groningen, USZ = University of Zurich.

**Figure 4:**
Bar graphs depict iterative development of consensus on the validity of reference values for radiomics features. We tried to find reliable reference values for radiomics features in an iterative standardization process. In phase I, features were computed without prior image processing, whereas in phase II, features were assessed after image processing with five predefined configurations (configurations A–E; Appendix E1 [online]). The panels show, A, the overall development of consensus on the validity of (tentative) reference values in phases I and II and, B, the development of consensus in phase II, according to image processing configuration. Consensus on the validity of a reference value is based on the number of research teams that produce the same value for a feature (weak: ≤3; moderate: three to five; strong: six to nine; very strong: ≥10). We analyzed consensus at each of the analysis time points, the time between which was variable (arbitrary unit; arb. unit). New features were included at time points 5 and 22, causing an apparent decrease in consensus. For phase II, we first analyzed consensus at time point 10. Image processing configurations C and D were altered after time point 16. Configuration E was altered after revising the resegmentation processing step at time point 22. See Appendix E1 (online) for more information regarding the timeline.

**Figure 5:**
Bar graph shows reproducibility of standardized radiomics features. We assessed reproducibility of 169 standardized features on a validation cohort of 51 patients with soft-tissue sarcoma using multimodality imaging (CT, fluorine 18 fluorodeoxyglucose PET, and T1-weighted MRI; shown as CT, PET and MRI) according to the feature values computed by research teams. We assigned each feature to a reproducibility category based on the lower boundary of the 95% confidence interval of the two-way random effects, single rater, absolute agreement intraclass correlation coefficient of the feature (poor: <0.50; moderate: 0.50–0.75; good: 0.75–0.90; excellent: ≥0.90). Five features could not be standardized in this study. Two features with unknown reproducibility were computed by fewer than two teams during validation.

See this image and copyright information in PMC

Comment in

The Long Route to Standardized Radiomics: Unraveling the Knot from the End.
Kuhl CK, Truhn D. Kuhl CK, et al. Radiology. 2020 May;295(2):339-341. doi: 10.1148/radiol.2020200059. Epub 2020 Mar 10. Radiology. 2020. PMID: 32159452 No abstract available.

References

1. La Thangue NB, Kerr DJ. . Predictive biomarkers: A paradigm shift towards personalized cancer medicine . Nat Rev Clin Oncol 2011. ; 8 ( 10 ): 587 – 596 . - PubMed
1. O’Connor JPB, Aboagye EO, Adams JE, et al. . Imaging biomarker roadmap for cancer studies . Nat Rev Clin Oncol 2017. ; 14 ( 3 ): 169 – 186 . - PMC - PubMed
1. Lambin P, Leijenaar RTH, Deist TM, et al. . Radiomics: The bridge between medical imaging and personalized medicine . Nat Rev Clin Oncol 2017. ; 14 ( 12 ): 749 – 762 . - PubMed
1. Morin O, Vallières M, Jochems A, et al. . A Deep Look into the Future of Quantitative Imaging in Oncology: A Statement of Working Principles and Proposal for Change . Int J Radiat Oncol Biol Phys 2018. ; 102 ( 4 ): 1074 – 1082 . - PubMed
1. Gillies RJ, Kinahan PE, Hricak H. . Radiomics: Images Are More than Pictures, They Are Data . Radiology 2016. ; 278 ( 2 ): 563 – 577 . - PMC - PubMed

Publication types

Actions
Actions

MeSH terms

Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions
Actions

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

The Image Biomarker Standardization Initiative: Standardized Quantitative Radiomics for High-Throughput Image-based Phenotyping

Affiliation

The Image Biomarker Standardization Initiative: Standardized Quantitative Radiomics for High-Throughput Image-based Phenotyping

Authors

Affiliation

Abstract

Figures

Comment in

References

Publication types

MeSH terms

Substances

Grants and funding

LinkOut - more resources

Full Text Sources

Other Literature Sources

Medical