ONEST (Observers Needed to Evaluate Subjective Tests) Analysis of Stromal Tumour-Infiltrating Lymphocytes (sTILs) in Breast Cancer and Its Limitations

Abstract

Tumour-infiltrating lymphocytes (TILs) reflect antitumour immunity. Their evaluation of histopathology specimens is influenced by several factors and is subject to issues of reproducibility. ONEST (Observers Needed to Evaluate Subjective Tests) helps in determining the number of observers that would be sufficient for the reliable estimation of inter-observer agreement of TIL categorisation. This has not been explored previously in relation to TILs. ONEST analyses, using an open-source software developed by the first author, were performed on TIL quantification in breast cancers taken from two previous studies. These were one reproducibility study involving 49 breast cancers, 23 in the first circulation and 14 pathologists in the second circulation, and one study involving 100 cases and 9 pathologists. In addition to the estimates of the number of observers required, other factors influencing the results of ONEST were examined. The analyses reveal that between six and nine observers (range 2-11) are most commonly needed to give a robust estimate of reproducibility. In addition, the number and experience of observers, the distribution of values around or away from the extremes, and outliers in the classification also influence the results. Due to the simplicity and the potentially relevant information it may give, we propose ONEST to be a part of new reproducibility analyses.

Keywords: European Working Group for Breast Screening Pathology; ONEST; TILs; breast cancer; international immuno-oncology biomarker working group; observers needed to evaluate subjective tests; reproducibility; sTILs; triple-negative; tumour-infiltrating lymphocytes.

Figure 1

ONEST plots of different cut-off values for 23 pathologists. (a) Full and (b) simplified ONEST plots for the 49 cases assessed by 23 pathologists for a cut-off of <50% vs. ≥50% sTILs. (c–e) Simplified ONEST plots for further cut-off values studied: (c) <60% vs. ≥60%; (d) <30% vs. ≥30%; and (e) <20%, 21–50%, >50%. Readings from the plots are included in Table 1. OPA (n = 23) values are the OPA values at the right side of the plots and reflect the proportion of cases with full agreement. ONEST values correspond to the number of observers on the x-axis, where the minimum curve levels off, and no substantial decrease is noted with further increase in the number of observers (this is highlighted by vertical segments between the x-axis value and the minimum curve). The bandwidth of the ONEST plot is visualised on the left side of the plot as the difference between the maximum and the minimum curves with 2 observers; this is the largest difference in agreement between two observers.

Figure 1

ONEST plots of different cut-off values for 23 pathologists. (a) Full and (b) simplified ONEST plots for the 49 cases assessed by 23 pathologists for a cut-off of <50% vs. ≥50% sTILs. (c–e) Simplified ONEST plots for further cut-off values studied: (c) <60% vs. ≥60%; (d) <30% vs. ≥30%; and (e) <20%, 21–50%, >50%. Readings from the plots are included in Table 1. OPA (n = 23) values are the OPA values at the right side of the plots and reflect the proportion of cases with full agreement. ONEST values correspond to the number of observers on the x-axis, where the minimum curve levels off, and no substantial decrease is noted with further increase in the number of observers (this is highlighted by vertical segments between the x-axis value and the minimum curve). The bandwidth of the ONEST plot is visualised on the left side of the plot as the difference between the maximum and the minimum curves with 2 observers; this is the largest difference in agreement between two observers.

Figure 1

ONEST plots of different cut-off values for 23 pathologists. (a) Full and (b) simplified ONEST plots for the 49 cases assessed by 23 pathologists for a cut-off of <50% vs. ≥50% sTILs. (c–e) Simplified ONEST plots for further cut-off values studied: (c) <60% vs. ≥60%; (d) <30% vs. ≥30%; and (e) <20%, 21–50%, >50%. Readings from the plots are included in Table 1. OPA (n = 23) values are the OPA values at the right side of the plots and reflect the proportion of cases with full agreement. ONEST values correspond to the number of observers on the x-axis, where the minimum curve levels off, and no substantial decrease is noted with further increase in the number of observers (this is highlighted by vertical segments between the x-axis value and the minimum curve). The bandwidth of the ONEST plot is visualised on the left side of the plot as the difference between the maximum and the minimum curves with 2 observers; this is the largest difference in agreement between two observers.

Figure 2

Partly overlapping simplified ONEST plots of 3 randomly selected 100 permutations (A, B, and C) for the <50% sTIL or more classification in C1 circulation without the two divergent classifiers; this example showed the lowest p-value in the Kruskal–Wallis test. Note: the y-axis only represents values between 0.5 and 1; despite not being statistically significantly different, the 3 randomly selected ONEST plots of 100 permutations yield 3 different ONEST values: 7 (A), 9 (B), and 5 (C) (to ease reading of the values, these are highlighted by vertical dashed segments between the x-axis value and the minimum curves), whereas the bandwidth is very similar (0.245 A, 0.265 B and C), and by definition, the OPA(21) value is identical (0.612). MAX: maximum curve; MED: median curve; MIN: minimum curve.