A scoping review and evidence gap analysis of clinical AI fairness

Abstract

The ethical integration of artificial intelligence (AI) in healthcare necessitates addressing fairness. AI fairness involves mitigating biases in AI and leveraging AI to promote equity. Despite advancements, significant disconnects persist between technical solutions and clinical applications. Through evidence gap analysis, this review systematically pinpoints the gaps at the intersection of healthcare contexts-including medical fields, healthcare datasets, and bias-relevant attributes (e.g., gender/sex)-and AI fairness techniques for bias detection, evaluation, and mitigation. We highlight the scarcity of AI fairness research in medical domains, the narrow focus on bias-relevant attributes, the dominance of group fairness centering on model performance equality, and the limited integration of clinician-in-the-loop to improve AI fairness. To bridge the gaps, we propose actionable strategies for future research to accelerate the development of AI fairness in healthcare, ultimately advancing equitable healthcare delivery.

Fig. 1

PRISMA-ScR flow diagram.

Fig. 2. The evidence gap analysis of AI fairness methodology developments and applications in cross-tabulation between medical fields and data types, bias-relevant attributes, and public dataset utilization.

Each unit (“1”) represents a single paper, where one paper may use multiple datasets, and each dataset can encompass various specialties, data types, and bias-relevant attributes. Papers were classified as “Public data only” if all datasets used were public; otherwise, they were classified as “Used own data“. CC Critical Care, ED Emergency Department, ID Infectious Diseases.

Fig. 3. The evidence gap analysis of fairness metrics across fairness notions and data characteristics.

The evidence gap analysis of fairness metrics, cross-tabulated by high-level fairness notions (group fairness, individual fairness and distribution fairness) against a data types and b bias-relevant attributes. Each unit (“1”) represents a single paper. One paper may involve multiple fairness notions and metrics.

Fig. 4. The evidence gap analysis of fairness methods and roles of bias-relevant attributes in cross-tabulation with algorithm types.

Each unit (“1”) represents a single paper. One study might contain several types of bias mitigation methods, and the bias-relevant attributes could have different roles in bias mitigation.