Prioritizing cases from a multi-institutional cohort for a dataset of pathologist annotations

Abstract

Objective: With the increasing energy surrounding the development of artificial intelligence and machine learning (AI/ML) models, the use of the same external validation dataset by various developers allows for a direct comparison of model performance. Through our High Throughput Truthing project, we are creating a validation dataset for AI/ML models trained in the assessment of stromal tumor-infiltrating lymphocytes (sTILs) in triple negative breast cancer (TNBC).

Materials and methods: We obtained clinical metadata for hematoxylin and eosin-stained glass slides and corresponding scanned whole slide images (WSIs) of TNBC core biopsies from two US academic medical centers. We selected regions of interest (ROIs) from the WSIs to target regions with various tissue morphologies and sTILs densities. Given the selected ROIs, we implemented a hierarchical rank-sort method for case prioritization.

Results: We received 122 glass slides and clinical metadata on 105 unique patients with TNBC. All received cases were female, and the mean age was 63.44 years. 60% of all cases were White patients, and 38.1% were Black or African American. After case prioritization, the skewness of the sTILs density distribution improved from 0.60 to 0.46 with a corresponding increase in the entropy of the sTILs density bins from 1.20 to 1.24. We retained cases with less prevalent metadata elements.

Conclusion: This method allows us to prioritize underrepresented subgroups based on important clinical factors. In this manuscript, we discuss how we sourced the clinical metadata, selected ROIs, and developed our approach to prioritizing cases for inclusion in our pivotal study.

Keywords: Data; Prioritization; Sampling; Validation.

Fig. 1

Institutional workflows for cohort identification. A. (Emory University Hospital) Patients identified through manual chart review beginning with an institutional breast cancer database. Provided database only included cases of HER2 0–1+ (negative) and HER2 2+ (equivocal). B. (Stony Brook University Hospital) Patients identified from Anatomic Pathology Database using natural language queries. The blue background denotes steps performed using the institution's pathology information system: CoPath used in both institutions. The yellow background denotes steps involving the electronic health record: Powerchart and Epic used at Emory. “n” refers to the count of unique patients. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Fig. 2

Clinical metadata distributions. A. Age distribution (decade binned) by Race. B. Distribution of age (decade binned) by Breast Cancer Stage. C. Distribution of breast cancer stages by Race.

Fig. 3

Results of hierarchical rank-sort method. A) Number of ROIs within each sTILs density bin for all 55 WSIs. Blue indicates ROIs included in the pivotal study batches by the hierarchical rank-sort method. Hashed regions represent all ROIs within that sTILs density bin including those not included in the pivotal study batches. B) Distribution of sTILs densities bins across pivotal study batches after sorting. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

Fig. 4

Characteristics of selected cases (WSIs). This figure shows the unique distribution of the patient demographic features (age, race, and ethnicity) and disease stages included in the pivotal study batches by the hierarchical rank-sort method. Our approach retained the patients within the lowest frequency age bins and disease stage, all “Black or African American” patients (13/47), and all “Hispanic or Latino” patients (5/47).

Fig. 5

Distribution of Pitfalls in the full set of ROIs selected and annotated and in the pivotal study batches selected by the hierarchical rank-sort method.