Comparison of Manual Versus QuPath Software-based Immunohistochemical Scoring Using Oral Squamous Cell Carcinoma as a Model

Abstract

Gold standard for immunohistochemical analyses is the manual assessment by two specialist pathologists. This process is time-consuming, highly dependent on the respective evaluator and often difficult to reproduce. The use of image analysis software, such as ImageJ, QuPath, or CellProfiler, which employ machine learning and/or deep learning mechanisms to perform biomarker analyses, offers a potential solution to these problems. The objective of our study is to evaluate whether digital assessment using the open-source software QuPath is comparable to manual evaluation and to examine the inter-evaluator variability between the two manual evaluators and two software-based evaluations. Six tissue microarrays (TMAs) were constructed for a cohort of 309 patients with primary oral squamous cell carcinoma (OSCC). The tumor tissue and corresponding non-lesional squamous epithelial mucosa specimen were immunohistochemically stained for the biomarkers Ki67, as a nuclear marker; the epidermal growth factor receptor (EGF-R), as a membranous marker; and the major histocompatibility complex class I (MHC-I) heavy chain (HC) expressed on the membrane and in the cytoplasm. The staining pattern was analyzed by two experienced, independent manual evaluators and by QuPath. The percentage of positive cells, for Ki67, and the histoscore (H-score) based on the percentage of positive cells and their staining intensity, for EGF-R and MHC-I, were determined as final values. The results yielded high to excellent spearman correlation coefficients for all three biomarkers (p<0.001) in lesional and non-lesional tissues. The Bland-Altman plots demonstrated a high degree of agreement between manual and software-based analysis, as well as inter-evaluator variability demonstrating a high comparability of the evaluation methods. However, a prerequisite for a proper software-based analysis is an accurate, time-consuming annotation of the single specimen, which requires users with a comprehensive understanding of histology and extensive training in QuPath. Once these requirements are met, the software-based analysis offers advantages for large-scale biomarker studies due to objective and reproducible comparability of the stainings leading to a greater accuracy as well as the reuse of established conditions across similar analyses without requiring further operator input.

Keywords: biomarker; digital pathology; head and neck squamous cell carcinoma; immunohistochemistry.

Figure 1.

QuPath: Cell segmentation steps. (A) Exemplary unprocessed TMA core with Ki67 staining. (B) Detection of all cells on the core with the command “cell detection.” (C) Segmentation of individual cells as tumor/squamous epithelium and stroma; tumoral/epithelial cells are red, stromal cells are green. (D) Classifying positive (red and dark green) and negative (blue and light green) stained cells with the command “Set cell intense classification.” Magnification 1:40 and 1:100; scale bars: 100 µm.

Figure 2.

Correction of artifacts. The image shows a staining artifact (A) before removing QuPath detected cells within the artifact and (B) after manually removing cells. Magnification 1:100; scale bar: 100 µm.

Figure 3.

Core level: Bland–Altman plots. Bland–Altman plots of agreement between manual and software-based scores (A1–A3, B1–B3), interrater agreement (A4–A6, B4–B6) in manual evaluation and interrater agreement in QuPath evaluation (A7–A9, B7–B9) at the core level for tumor tissue (A) and normal squamous epithelium (B). X-axis shows the average evaluation score between the assessment methods or the two manuals as well as QuPath evaluators. Mean differences and limits of agreement (1.96 × standard deviation ≙ 95% confidence interval) are presented on y-axis.

Figure 4.

Different quality of membrane detection in QuPath. Representative images of EGF-R-stained OSCC with (A, B) a case with well-detected cell membranes and (C, D) a case with poorly detected cell membranes. In (A, C), the unannotated scans are shown; in (B, D), QuPath annotations are shown. Colors of detected cells correspond to the staining intensity (blue = negative, yellow = weak, orange = moderate, red = strong, green = stroma). Magnification 1:200; scale bar: 50 µm.

Appendix Figure A1.

QuPath: estimate stain vectors. QuPath generates 2D scatterplots to show the relationship of blue, red, and green values for each pixel. The software calculates the optimal stain vectors (including the majority of the scattered points) for hematoxylin and DAB with consideration given to the background.

Appendix Figure A2.

Bland–Altman plots of agreement between manual and software-based scores (A1–A3, B1–B3), interrater agreement (A4–A6, B4–B6) in manual evaluation and interrater agreement (A7–A9, B7–B9) in QuPath evaluation at case level for tumor tissue (A) and normal squamous epithelium (B). X-axis shows the average evaluation score between the assessment methods or the two manuals as well as QuPath evaluators at case level. Mean differences and limits of agreement (1.96 × standard deviation ≙ 95% confidence interval) are presented on y-axis. Abbreviations: EGF-R, epidermal growth factor receptor; MHC-I HC, major histocompatibility complex class I heavy chain.