Assessing fairness in machine learning models: A study of racial bias using matched counterparts in mortality prediction for patients with chronic diseases

Abstract

Objective: Existing approaches to fairness evaluation often overlook systematic differences in the social determinants of health, like demographics and socioeconomics, among comparison groups, potentially leading to inaccurate or even contradictory conclusions. This study aims to evaluate racial disparities in predicting mortality among patients with chronic diseases using a fairness detection method that considers systematic differences.

Methods: We created five datasets from Mass General Brigham's electronic health records (EHR), each focusing on a different chronic condition: congestive heart failure (CHF), chronic kidney disease (CKD), chronic obstructive pulmonary disease (COPD), chronic liver disease (CLD), and dementia. For each dataset, we developed separate machine learning models to predict 1-year mortality and examined racial disparities by comparing prediction performances between Black and White individuals. We compared racial fairness evaluation between the overall Black and White individuals versus their counterparts who were Black and matched White individuals identified by propensity score matching, where the systematic differences were mitigated.

Results: We identified significant differences between Black and White individuals in age, gender, marital status, education level, smoking status, health insurance type, body mass index, and Charlson comorbidity index (p-value < 0.001). When examining matched Black and White subpopulations identified through propensity score matching, significant differences between particular covariates existed. We observed weaker significance levels in the CHF cohort for insurance type (p = 0.043), in the CKD cohort for insurance type (p = 0.005) and education level (p = 0.016), and in the dementia cohort for body mass index (p = 0.041); with no significant differences for other covariates. When examining mortality prediction models across the five study cohorts, we conducted a comparison of fairness evaluations before and after mitigating systematic differences. We revealed significant differences in the CHF cohort with p-values of 0.021 and 0.001 in terms of F1 measure and Sensitivity for the AdaBoost model, and p-values of 0.014 and 0.003 in terms of F1 measure and Sensitivity for the MLP model, respectively.

Discussion and conclusion: This study contributes to research on fairness assessment by focusing on the examination of systematic disparities and underscores the potential for revealing racial bias in machine learning models used in clinical settings.

Keywords: Chronic Disease; Electronic Health Records; Fairness Analysis; Machine Learning; Mortality Prediction; Racism.

Figure 1.

Training-validation data splitting method with counterparts. The total population is initially divided into three distinct parts: Black, White-matched, and Others. The Black and White-matched groups serve as counterparts. The “Others” group comprises the remaining population, including non-matched White individuals and those of races other than Black or White. The splitting method for the matched White population mirrors that of the Black population to maintain counterpart matching. The “Others” group is splitted using randomization. These segments, labeled as B1, M1 and O1, are concatenated to form the final training set.

Figure 2.

Performance comparison on Black, White-matched, White-total, and the total population. Five datasets are studied, each focusing on a different chronic condition: congestive heart failure (CHF), chronic kidney disease (CKD), chronic obstructive pulmonary disease (COPD), chronic liver disease (CLD), and dementia. For each chronic disease group, four types of 1-year mortality prediction models are compared, including Logistic Regression (LR), Random Forest (RF), AdaBoost (Ada), and Multilayer Perceptron (MLP). The model performance is assessed using sensitivity, F1 score, and AUROC.

Figure 3.

Comparison of fairness evaluation methods on CHF and Dementia Cohorts. For each type of disease, two models (Ada and MLP) and two fairness assessments (${\mathrm{\Delta }}_{f1}$ and ${\mathrm{\Delta }}_{Se}$) are considered. Each scatter plot compares the results of fairness evaluation between counterparts (i.e., Black vs White-matched; x-axis) and fairness evaluation between overall groups (i.e., Black vs. White-total; y-axis) from five repeated experiments. Each point on the scatter plot represents a pair of results from the same experiment. The p-values are paired t-tests that are conducted to assess the significance of differences between two fairness evaluation methods.