Accelerating pharmaceutical R&D with a user-friendly AI system for histopathology image analysis

Abstract

A system for analysis of histopathology data within a pharmaceutical R&D environment has been developed with the intention of enabling interdisciplinary collaboration. State-of-the-art AI tools have been deployed as easy-to-use self-service modules within an open-source whole slide image viewing platform, so that non-data scientist users (e.g., clinicians) can utilize and evaluate pre-trained algorithms and retrieve quantitative results. The outputs of analysis are automatically cataloged in the database to track data provenance and can be viewed interactively on the slide as annotations or heatmaps. Commonly used models for analysis of whole slide images including segmentation, extraction of hand-engineered features for segmented regions, and slide-level classification using multi-instance learning are included and new models can be added as needed. The source code that supports running inference with these models internally is backed up by a robust CI/CD pipeline to ensure model versioning, robust testing, and seamless deployment of the latest models. Examples of the use of this system in a pharmaceutical development workflow include glomeruli segmentation, enumeration of podocyte count from WT-1 immuno-histochemistry, measurement of beta-1 integrin target engagement from immunofluorescence, digital glomerular phenotyping from periodic acid-Schiff histology, PD-L1 score prediction using multi-instance learning, and the deployment of the open-source Segment Anything model to speed up annotation.

Keywords: Annotation; Model cataloging; Segment Anything; Segmentation; Visualization.

Fig. 1

Overview of proposed system for histopathology analysis. The system is centered around a cloud-based histology viewer (DSA) which is run on AWS EC2. The DSA interface acts as a system for exploring the database and associated metadata, as well as tracking data provenance. HistomicsUI (a component of DSA) allows viewing and annotation of histopathology data by users. The DSA queries a series of external S3 buckets which store the data. Data queries and access pass through a system for data stewardship ensuring proper management and governance of data. External compute resources are connected to the system allowing data scientists to use annotations to create models using a shared codebase which is continuously improved. Trained models can be deployed and run through the DSA interface, where the results are automatically cataloged.

Fig. 2

Panel A shows loading times for 10 WSIs by storage location. This reflects the time to open a slide and zoom into the maximum magnification level. Data stored in EC2 is located on the instance, which is hosting the DSA, S3 data is stored on a remote S3 bucket. Both EC2 and S3 are accessed via the DSA over web. Data stored locally is opened via Aperio Imagescope on a user’s local computer. Panels B–D show examples of the user interface of the developed tool. Panel B depicts a folder of data in the system. Thumbnails of slides in this folder are pictured and the metadata fields associated with each slide can be configured, searched, and sorted. Panel C depicts the HistomicsUI slide viewer, where users can interact with the data, annotate, or submit analysis jobs using deployed algorithms. Here, an algorithm for PD-L1 scoring using multi-instance learning is shown, but a model agnostic version is also available using the internally developed codebase. Panel D depicts a deployed segmentation algorithm which can also be run through the user interface. The model parameters for this segmentation algorithm are user selectable which makes it reusable for multiple tissue types.

Fig. 3

Examples of computationally produced annotations in the DSA. Panel A shows glomeruli segmentation (blue) and β1 integrin detection (orange) from immunofluorescence-stained kidney tissue. Panel B shows glomeruli segmentation (blue) and podocyte detection (orange) from renal tissue stained using Wilms’ tumor-1 immunohistochemistry. Panel C depicts multi-compartment instance segmentation of renal tissue, tubules (blue), glomeruli (yellow), sclerotic glomeruli (red), and arteries (orange). Panel D depicts various heatmaps of attention scores for a multi-instance learning network trained to predict PD-L1 score on H&E tissue sections.

Fig. 4

Tools for speeding annotation using zero shot learning. We have deployed a foundational model for segmentation (Segment Anything) to speed up annotation of structures in WSIs. Panel A depicts using the Segment Anything model for pre-segmentation of the entire WSI. Note the slide is automatically tiled to fit into memory, and the magnification of the tiles is user selectable through the UI. Panel B shows the ability of a user to right click and assign detected contours labels from a pre-defined list which can be set on a folder level. Panel C shows the ability to run the Segment Anything models on user defined regions of interest. This is similar to pre segmentation of the entire WSI but is useful if the user only wants to annotate specific sections of the slide. Finally, panel D shows the ability to generate segmentations from user prompts. Here, a user roughly annotates structures of interest by placing a bounding box around them which is converted to a segmentation boundary using the Segment Anything model.