Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2024 Jul 12;3(7):e0000542.
doi: 10.1371/journal.pdig.0000542. eCollection 2024 Jul.

Variable importance analysis with interpretable machine learning for fair risk prediction

Yilin Ning  1 , Siqi Li  1 , Yih Yng Ng  2   3 , Michael Yih Chong Chia  4 , Han Nee Gan  5 , Ling Tiah  5 , Desmond Renhao Mao  6 , Wei Ming Ng  7 , Benjamin Sieu-Hon Leong  8 , Nausheen Doctor  9 , Marcus Eng Hock Ong  10   11   12 , Nan Liu  1   10   13
Affiliations

Variable importance analysis with interpretable machine learning for fair risk prediction

Yilin Ning et al. PLOS Digit Health. .

Abstract

Machine learning (ML) methods are increasingly used to assess variable importance, but such black box models lack stability when limited in sample sizes, and do not formally indicate non-important factors. The Shapley variable importance cloud (ShapleyVIC) addresses these limitations by assessing variable importance from an ensemble of regression models, which enhances robustness while maintaining interpretability, and estimates uncertainty of overall importance to formally test its significance. In a clinical study, ShapleyVIC reasonably identified important variables when the random forest and XGBoost failed to, and generally reproduced the findings from smaller subsamples (n = 2500 and 500) when statistical power of the logistic regression became attenuated. Moreover, ShapleyVIC reasonably estimated non-significant importance of race to justify its exclusion from the final prediction model, as opposed to the race-dependent model from the conventional stepwise model building. Hence, ShapleyVIC is robust and interpretable for variable importance assessment, with potential contribution to fairer clinical risk prediction.

PubMed Disclaimer

Conflict of interest statement

MEH Ong reports an advisory relationship with Global Healthcare SG, a commercial entity that manufactures cooling devices. MEH Ong has a licensing agreement and a patent filed (Application no: 13/047,348) with ZOLL Medical Corporation for a study titled "Method of predicting acute cardiopulmonary events and survivability of a patient". All other authors have no conflict of interests to declare.

Figures

Fig 1
Fig 1. Variable importance analysis from ShapleyVIC (arranged by overall importance) and logistic regression analyses (arranged in ascending order of p-value), from the main analysis (n = 4494) and simulated experiments (n = 2500 and n = 500).
Blue color and “*” indicate significant variable importance, i.e., 95% prediction interval above zero for ShapleyVIC and p-value<0.05 for logistic regression.
Fig 2
Fig 2. Variable importance analysis from ShapleyVIC and logistic regression analyses in 3 additional simulation experiments with n = 500, arranged by the same variable ordering as in Fig 1.
Blue color and “*” indicate significant variable importance, i.e., 95% prediction interval above zero for ShapleyVIC and p-value<0.05 for logistic regression.
Fig 3
Fig 3
Variable importance based on (A) random forest and (B) XGBoost from the main analysis.
See this image and copyright information in PMC

References

    1. Berdowski J, Berg RA, Tijssen JGP, Koster RW. Global incidences of out-of-hospital cardiac arrest and survival rates: systematic review of 67 prospective studies. Resuscitation. 2010;81: 1479–1487. doi: 10.1016/j.resuscitation.2010.08.006 - DOI - PubMed
    1. Feng D, Li C, Yang X, Wang L. Gender differences and survival after an out-of-hospital cardiac arrest: a systematic review and meta-analysis. Intern Emerg Med. 2021;16: 765–775. doi: 10.1007/s11739-020-02552-4 - DOI - PubMed
    1. Lei H, Hu J, Liu L, Xu D. Sex differences in survival after out-of-hospital cardiac arrest: a meta-analysis. Crit Care Lond Engl. 2020;24: 613. doi: 10.1186/s13054-020-03331-5 - DOI - PMC - PubMed
    1. Rob D, Kavalkova P, Smalcova J, Franek O, Smid O, Komarek A, et al.. Gender differences and survival after out of hospital cardiac arrest. Am J Emerg Med. 2022;55: 27–31. doi: 10.1016/j.ajem.2022.02.032 - DOI - PubMed
    1. Kotini-Shah P, Del Rios M, Khosla S, Pugach O, Vellano K, McNally B, et al.. Sex differences in outcomes for out-of-hospital cardiac arrest in the United States. Resuscitation. 2021;163: 6–13. doi: 10.1016/j.resuscitation.2021.03.020 - DOI - PMC - PubMed

LinkOut - more resources