Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
Multicenter Study
. 2025 Jul 1;8(7):e2518906.
doi: 10.1001/jamanetworkopen.2025.18906.

Pathologist-Read vs AI-Driven Assessment of Tumor-Infiltrating Lymphocytes in Melanoma

Thazin N Aung  1 , Matthew Liu  1 , David Su  2 , Saba Shafi  3 , Ceren Boyaci  4   5 , Sanna Steen  4   5 , Nikolaos Tsiknakis  5 , Joan Martinez Vidal  5   6 , Nigel Maher  7 , Goran Micevic  2 , Samuel X Tan  8 , Matthew D Vesely  2 , Saeed Nourmohammadi  9 , Yalai Bai  1 , Dijana Djureinovic  2 , Pok Fai Wong  1 , Katherine Bates  1 , Nay N N Chan  1 , Niki Gavirelatou  1 , Mengni He  1 , Sneha Burela  1 , Robert Barna  10 , Martina Bosic  11 , Konstantin Bräutigam  12 , Irineu Illabochaca  13 , Zhou Chenhao  8 , Joao Gama  14 , Bianca Kreis  15 , Reka Mohacsi  16 , Nir Pillar  17 , Joao Pinto  18 , Christos Poulios  19 , Maria Angeliki Toli  5 , Evangelos Tzoras  5 , Yadriel Bracero  20 , Francesca Bosisio  21 , Gábor Cserni  22 , Alis Dema  10 , Francesco Fortarezza  23 , Mercedes Solorzano Gonzalez  24 , Irene Gullo  25   26   27 , Francisco Javier Queipo Gutiérrez  24 , Ezgi Hacihasanoglu  28 , Viktor Jovic  4 , Bianca Lazar  29 , Maria Olinca  30 , Christina Neppl  31 , Rui Caetano Oliveira  14 , Federica Pezzuto  32 , Daniel Gomes Pinto  33 , Vanda Plotar  34 , Ovidiu Pop  35 , Tilman Rau  31 , Kristijan Skok  36   37 , Wenwen Sun  4   5 , Ezgi Dicle Serbes  38   39 , Wiebke Solass  40 , Olga Stanowska  40 , Marcell Szasz  16 , Krzysztof Szymonski  41 , Franziska Thimm  4 , Danielle Vignati  7 , Alon Vigdorovits  35 , Victor Prieto  42   43 , Tobias Sinnberg  44   45 , James Wilmott  7 , Shawn Cowper  2 , Jonathan Warrell  46   47 , Yvonne Saenger  20 , Johan Hartman  4   5 , Jasmine Plummer  48 , Iman Osman  13 , David L Rimm  1   2 , Balazs Acs  4   5
Affiliations
Multicenter Study

Pathologist-Read vs AI-Driven Assessment of Tumor-Infiltrating Lymphocytes in Melanoma

Thazin N Aung et al. JAMA Netw Open. .

Abstract

Importance: Tumor-infiltrating lymphocytes (TILs) are a provocative biomarker in melanoma, influencing diagnosis, prognosis, and immunotherapy outcomes; however, traditional pathologist-read TIL assessment on hematoxylin and eosin-stained slides is prone to interobserver variability, leading to inconsistent clinical decisions. Therefore, development of newer TIL scoring approaches that produce more reliable and consistent readouts is important.

Objective: To evaluate the analytical and clinical validity of a machine learning algorithm for TIL quantification in melanoma compared with traditional pathologist-read methods.

Design, setting, and participants: This multioperator, global, multi-institutional prognostic study compared TIL scoring reproducibility between traditional pathologist-read methods and an artificial intelligence (AI)-driven approach. The study was conducted using retrospective cohorts of patients with melanoma between January 2022 and June 2023 across 45 institutions, with tissue evaluated by participants from academic, clinical, and research institutions. Participants were selected to ensure diverse expertise and professional backgrounds.

Main outcomes and measures: Intraclass correlation coefficient (ICC) values were calculated for the manual and AI-assisted arms using log-transformed data. Kendall W values were calculated for Clark scores (brisk = 3, nonbrisk = 2, and sparse = 1). Reliabilities of ICC and W values were classified as moderate (0.40-0.60), good (0.61-0.80), or excellent (>0.80). AI TIL measurements were dichotomized using the 16.6 and median cutoffs. Univariable and multivariable Cox regression analyses assessed the prognostic value of TIL scores adjusted for clinicopathologic variables.

Results: There were 111 patients with melanoma in the independent testing cohort (median [range] age at diagnosis, 61.0 [25.0-87.0] years; 56 [50.5%] male) who contributed melanoma whole tissue sections. A total of 98 participants evaluated TILs on 60 hematoxylin and eosin-stained melanoma tissue sections. All 40 participants in the manual arm were pathologists, while the AI-assisted arm included 11 pathologists and 47 nonpathologists (scientists). The AI algorithm demonstrated superior reproducibility, with ICCs higher than 0.90 for all machine learning TIL variables, significantly outperforming manual assessments (ICC, 0.61 for AI-derived stromal TILs vs Kendall W, 0.44 for manual Clark TIL scoring). AI-based TIL scores showed prognostic associations with patient outcomes (n = 111) using the median cutoff approach with a hazard ratio (HR) of 0.45 (95% CI, 0.26-0.80; P = .005), and using the cutoff of 16.6, with an HR of 0.56 (95% CI, 0.32-0.98; P = .04).

Conclusions and relevance: In this prognostic study of TIL quantification in melanoma, the AI algorithm demonstrated superior reproducibility and prognostic associations compared with traditional methods. Although the retrospective nature of the cohorts limits demonstration of clinical utility, the publicly available dataset and open-source AI tool offer a foundation for future validation and integration into melanoma management.

PubMed Disclaimer

Conflict of interest statement

Conflict of Interest Disclosures: Dr Aung reported receiving support from a Robert E. Leet and Clara Guthrie Patterson Trust Mentored Research Award (Bank of America, Private Bank, Trustee), the Lion Heart Breast Cancer Research Foundation, and the Tower Cancer Research Foundation. Dr Wong reported receiving personal fees from AbbVie Inc and Verily Life Sciences LLC outside the submitted work. Dr Bosisio reported receiving grants from Research Foundation Flanders (FWO) outside the submitted work. Dr Gomes Pinto reported receiving personal fees from AstraZeneca Portugal, Roche Portugal, MSD Portugal, and Hologic Iberia; and receiving nonfinancial support from Daiichi Sankyo Portugal outside the submitted work. Dr Szasz reported receiving grants from the Hungarian Scientific Research Fund (OTKA). Dr Prieto reported being a consultant for Merck, Orlucent, and Castle Biosciences outside the submitted work. Dr Saenger reported receiving personal fees from Regeneron outside the submitted work. Dr Hartman reported receiving grants from the Swedish Cancer Fund and from Region Stockholm during the conduct of the study; and receiving personal fees from Stratipath outside the submitted work. Dr Plummer reported receiving grants from the Ovarian Cancer Research Alliance and the Chan Zuckerburg Foundation; and receiving travel support from DAVA Oncology, PMLS and Advancing Precision Medicine. Dr Rimm reported receiving grants from Yale University SPORE Developmental Research Program during the conduct of the study; receiving personal fees from AstraZeneca, Cell Signaling Technology, Cepheid, Danaher, Daiichi Sankyo, Halda Biotherapeutics, Incendia, NextCure, Nucleai, Paige AI, Regeneron, and Sanofi; and receiving grants from Cepheid, Navigate Biopharma, NextCure, Konica Minolta, Leica/Danaher, and Lunit outside the submitted work. Dr Acs reported receiving grants from The Swedish Society for Medical Research (Svenska Sällskapet för Medicinsk Forskning) during the conduct of the study; and being supported by Region Stockholm (clinical research appointment). No other disclosures were reported.

Figures

Figure 1.
Figure 1.. Diagram for the International Round-Robin Study
TIL represents tumor-infiltrating lymphocyte.
Figure 2.
Figure 2.. Interobserver Variability in AI-Assisted vs Manual Tumor-Infiltrating Lymphocyte (TIL) Scoring
Heatmaps and corresponding box plots display variability in selected TIL variables. (A) electronic TIL (eTILs) score from the AI arm demonstrating lower operator variability compared with (B) stromal TIL (sTILs) score from the manual arm. The vertical axes for each heatmap-box plot set are ordered by their respective median values. Concordance statistics are provided for each variable in both arms. CV indicates coefficient of variation. (C) Comparison of intraclass correlation coefficient (ICC) values (for electronic-area-stromal TIL [easTILs], eTILs, and sTILs scores) and Kendall W (for Clark TIL scoring) values between the AI arm and the manual pathologist-read TIL scoring arm.
Figure 3.
Figure 3.. Comparative Survival Analysis Based on Tumor-Infiltrating Lymphocyte (TIL) Scoring Methods
eTILs represents electronic TIL score; etTILs, electronic-total TIL score; sTILs, stromal TIL score; HR, hazard ratio.
See this image and copyright information in PMC

References

    1. Galon J, Costes A, Sanchez-Cabo F, et al. Type, density, and location of immune cells within human colorectal tumors predict clinical outcome. Science. 2006;313(5795):1960-1964. doi: 10.1126/science.1129139 - DOI - PubMed
    1. Sharma P, Shen Y, Wen S, et al. CD8 tumor-infiltrating lymphocytes are predictive of survival in muscle-invasive urothelial carcinoma. Proc Natl Acad Sci U S A. 2007;104(10):3967-3972. doi: 10.1073/pnas.0611618104 - DOI - PMC - PubMed
    1. Fridman WH, Pagès F, Sautès-Fridman C, Galon J. The immune contexture in human tumours: impact on clinical outcome. Nat Rev Cancer. 2012;12(4):298-306. doi: 10.1038/nrc3245 - DOI - PubMed
    1. Hodi FS, O’Day SJ, McDermott DF, et al. Improved survival with ipilimumab in patients with metastatic melanoma. N Engl J Med. 2010;363(8):711-723. doi: 10.1056/NEJMoa1003466 - DOI - PMC - PubMed
    1. Chatziioannou E, Roßner J, Aung TN, et al. Deep learning-based scoring of tumour-infiltrating lymphocytes is prognostic in primary melanoma and predictive to PD-1 checkpoint inhibition in melanoma metastases. EBioMedicine. 2023;93:104644. doi: 10.1016/j.ebiom.2023.104644 - DOI - PMC - PubMed

Publication types