A Distributed System Improves Inter-Observer and AI Concordance in Annotating Interstitial Fibrosis and Tubular Atrophy

Abstract

Histologic examination of interstitial fibrosis and tubular atrophy (IFTA) is critical to determine the extent of irreversible kidney injury in renal disease. The current clinical standard involves pathologist's visual assessment of IFTA, which is prone to inter-observer variability. To address this diagnostic variability, we designed two case studies (CSs), including seven pathologists, using HistomicsTK- a distributed system developed by Kitware Inc. (Clifton Park, NY). Twenty-five whole slide images (WSIs) were classified into a training set of 21 and a validation set of four. The training set was composed of seven unique subsets, each provided to an individual pathologist along with four common WSIs from the validation set. In CS 1, all pathologists individually annotated IFTA in their respective slides. These annotations were then used to train a deep learning algorithm to computationally segment IFTA. In CS 2, manual and computational annotations from CS 1 were first reviewed by the annotators to improve concordance of IFTA annotation. Both the manual and computational annotation processes were then repeated as in CS1. The inter-observer concordance in the validation set was measured by Krippendorff's alpha (KA). The KA for the seven pathologists in CS1 was 0.62 with CI [0.57, 0.67], and after reviewing each other's annotations in CS2, 0.66 with CI [0.60, 0.72]. The respective CS1 and CS2 KA were 0.58 with CI [0.52, 0.64] and 0.63 with CI [0.56, 0.69] when including the deep learner as an eighth annotator. These results suggest that our designed annotation framework refines agreement of spatial annotation of IFTA and demonstrates a human-AI approach to significantly improve the development of computational models.

Figure 1.. Overview of the distributed system:

(A) Using HistomicsTK to annotate IFTA via pathologists. (B) WSIs and associated annotations obtained from the pathologists. (C) Training set of WSIs and annotations were used to train HAIL. (D) Validation image and the corresponding predicted region. (E) Comparing validation image annotations from seven pathologists and HAIL by assigning different label colors in HistomicsTK.

Figure 2.. HistomicsTK and Digital Slide Archive (DSA) workflow.

(A) HistomicsTK viewer with basic operations, such as zoom, pan, and annotation. (B) Digital Slide Archive (DSA) interface which stores WSIs, annotation labels and meta data.

Figure 3.. Overview of the project design to annotate IFTA on WSIs.

WSIs are allocated along different annotators to label IFTA. Inside yellow circle, WSIs are considered for validation set and rest of the WSIs are considered for training set for training HAIL.