A method to reduce variability in scoring antibody-mediated rejection in renal allografts: implications for clinical trials - a retrospective study

Abstract

Poor reproducibility in scoring antibody-mediated rejection (ABMR) using the Banff criteria might limit the use of histology in clinical trials. We evaluated the reproducibility of Banff scoring of 67 biopsies by six renal pathologists at three institutions. Agreement by any two pathologists was poor: 44.8-65.7% for glomerulitis, 44.8-67.2% for peritubular capillaritis, and 53.7-80.6% for chronic glomerulopathy (cg). All pathologists agreed on cg0 (n = 20) and cg3 (n = 9) cases, however, many disagreed on scores of cg1 or cg2. The range for the incidence of composite diagnoses by individual pathologists was: 16.4-22.4% for no ABMR; 17.9-47.8% for active ABMR; and 35.8-59.7% for chronic, active antibody-mediated rejection (cABMR). A "majority rules" approach was then tested in which the scores of three pathologists were used to reach an agreement. This increased consensus both for individual scores (ex. 67.2-77.6% for cg) and for composite diagnoses (ex. 74.6-86.6% cABMR). Modeling using these results showed that differences in individual scoring could affect the outcome assessment in a mock study of cABMR. We conclude that the Banff schema has high variability and a majority rules approach could be used to adjudicate differences between pathologists and reduce variability in scoring in clinical trials.

Keywords: kidney clinical; rejection.

Figure 1.

Heatmaps for Banff scores using one-pathologist versus a 3 pathologist ‘consensus’ score.

Figure 2.

A flow diagram for diagnosis in a hypothetical clinical trial for cABMR involving 150 patients. The ground truth is given by the 6 pathologist consensus diagnosis and the range of diagnoses either in agreement or not is given for a three pathologist consensus or a single pathologist.

Figure 3.

A mock trial involving 150 patients. In the ideal situation a pathologist would always and only identify positive cases as positive for cABMR. Given the PPV and diagnostic agreement for one pathologist and a three pathologist ‘majority rules’, the conclusion of the trial could be affected.

Figure 4.

The effect of a lack of reproducibility versus A) effect size and B) sample size based on the above agreement and PPV. In order to produce lines, the mock trial was repeated for a variety of different effect sizes (odds ratios) and sample sizes.