Generalizability assessment of AI models across hospitals in a low-middle and high income country

Abstract

The integration of artificial intelligence (AI) into healthcare systems within low-middle income countries (LMICs) has emerged as a central focus for various initiatives aiming to improve healthcare access and delivery quality. In contrast to high-income countries (HICs), which often possess the resources and infrastructure to adopt innovative healthcare technologies, LMICs confront resource limitations such as insufficient funding, outdated infrastructure, limited digital data, and a shortage of technical expertise. Consequently, many algorithms initially trained on data from non-LMIC settings are now being employed in LMIC contexts. However, the effectiveness of these systems in LMICs can be compromised when the unique local contexts and requirements are not adequately considered. In this study, we evaluate the feasibility of utilizing models developed in the United Kingdom (a HIC) within hospitals in Vietnam (a LMIC). Consequently, we present and discuss practical methodologies aimed at improving model performance, emphasizing the critical importance of tailoring solutions to the distinct healthcare systems found in LMICs. Our findings emphasize the necessity for collaborative initiatives and solutions that are sensitive to the local context in order to effectively tackle the healthcare challenges that are unique to these regions.

Fig. 1. t-SNE plot of UK and Vietnam datasets with reduced feature set.

Plot includes all positive COVID-19 samples in UK and Vietnam datasets, including the matched/reduced set of features.

Fig. 2. t-SNE plot of UK and Vietnam datasets with comprehensive feature set.

Plot includes all positive COVID-19 samples in UK and Vietnam datasets, including the comprehensive set of features, which were generated using the GATS technique.

Fig. 3. COVID-19 diagnosis performance across logistic regression, XGBoost, and neural network models trained on the UK data.

Results are presented as AUROC for the reduced feature set and the comprehensive feature set (GATS-filled), with * representing the comprehensive dataset. Error bars are shown as 95% confidence intervals (CIs), which are computed using 1000 bootstrapped samples drawn from each test set. Source data are provided as a Source Data file.

Fig. 4. COVID-19 diagnosis AUROC performance at HTD and NHTD using neural network models which were ready-made (the UK-based models) and models which were fine-tuned using transfer learning.

Models trained and tested locally at HTD and NHTD are represented by the horizontal purple and yellow dotted lines, respectively. Results are presented for the reduced feature set and the comprehensive feature set (GATS-filled), with * representing the comprehensive dataset. Error bars are shown as 95% confidence intervals (CIs), which are computed using 1000 bootstrapped samples drawn from each test set. Source data are provided as a Source Data file. TL transfer learning.

Fig. 5. COVID-19 diagnosis AUPRC performance at HTD and NHTD using neural network models which were ready-made (the UK-based models) and models which were fine-tuned using transfer learning.

Models trained and tested locally at HTD and NHTD are represented by the horizontal purple and yellow dotted lines, respectively. Results are presented for the reduced feature set and the comprehensive feature set (GATS-filled), with * representing the comprehensive dataset. Error bars are shown as 95% confidence intervals (CIs), which are computed using 1000 bootstrapped samples drawn from each test set. Source data are provided as a Source Data file. TL Transfer Learning.