Image analysis and machine learning in digital pathology: Challenges and opportunities

Abstract

With the rise in whole slide scanner technology, large numbers of tissue slides are being scanned and represented and archived digitally. While digital pathology has substantial implications for telepathology, second opinions, and education there are also huge research opportunities in image computing with this new source of "big data". It is well known that there is fundamental prognostic data embedded in pathology images. The ability to mine "sub-visual" image features from digital pathology slide images, features that may not be visually discernible by a pathologist, offers the opportunity for better quantitative modeling of disease appearance and hence possibly improved prediction of disease aggressiveness and patient outcome. However the compelling opportunities in precision medicine offered by big digital pathology data come with their own set of computational challenges. Image analysis and computer assisted detection and diagnosis tools previously developed in the context of radiographic images are woefully inadequate to deal with the data density in high resolution digitized whole slide images. Additionally there has been recent substantial interest in combining and fusing radiologic imaging and proteomics and genomics based measurements with features extracted from digital pathology images for better prognostic prediction of disease aggressiveness and patient outcome. Again there is a paucity of powerful tools for combining disease specific features that manifest across multiple different length scales. The purpose of this review is to discuss developments in computational image analysis tools for predictive modeling of digital pathology images from a detection, segmentation, feature extraction, and tissue classification perspective. We discuss the emergence of new handcrafted feature approaches for improved predictive modeling of tissue appearance and also review the emergence of deep learning schemes for both object detection and tissue classification. We also briefly review some of the state of the art in fusion of radiology and pathology images and also combining digital pathology derived image measurements with molecular "omics" features for better predictive modeling. The review ends with a brief discussion of some of the technical and computational challenges to be overcome and reflects on future opportunities for the quantitation of histopathology.

Keywords: Deep learning; Digital pathology; Omics; Radiology.

Fig. 1

Results of nuclear detection using the locally constrained deep learning approach proposed by Sirinukunwattana et al. (2016).

Fig. 2

Examples of domain agnostic and domain inspired handcrafted features for disease characterization and outcome prediction. An example of domain agnostic cell cluster graph features (b) to capture the spatial architecture of nuclei in p16 + oropharyngeal cancers (a). These CCG features were shown to predict progression (Lewis et al., 2014) in these cancers more accurately than T-stage and lymph node status (c).

Fig. 3

Image-guided specimen slicing process. The first step is to select a desired slicing plane orientation and location from the 3D in vivo image. By using image-based registration, this plane is transformed to the space of the ex vivo specimen image. Corresponding fiducial markers are localized on the ex vivo image (by using software) and on the physical specimen (by using a tracked stylus), and a landmark transform aligning these fiducial markers transforms the slicing plane into the space of the tracker. The tracked stylus is used to direct the insertion of three slicing plane–defining pins into the specimen, which align the specimen in a slotted forceps for slicing. This permits the establishment of a correspondence between stained histologic slices and in vivo imaging planes, which are registered by using a non-rigid registration. Reproduced from (Ward et al., 2012).