Deep learning to predict microscope images

Abstract

A species of neural network first described in 2015 can be trained to translate between images of the same field of view acquired by different modalities. Trained networks can use information inherent in grayscale images of cells to predict fluorescent signals.

Fig 1.

a) Translating to predict. Top. Network in Panel c is trained on pairs of images of cells acquired under Modality M1 (eg. brightfield) and Modality M2 (here, fluorescence images of signal from a protein the nuclear envelope). Bottom. Given a new brightfield image, the trained network predicts the nuclear envelope signal from information in an image of a field of unlabelled cells. b) A network that classifies. A simple NN similar to Alexnet that identifies images of cats. Its operation shown in captions. In an untrained network, values of the filter matrices (which are convolved with upstream information to create feature maps) and of the weights of the individual inputs to the downstream “fully connected” “neurons” (affine operations and activation functions) are set randomly, and values of the filter matrices and input weights move toward optimum values during training. The network is trained by exposure to images, some of which are labeled “Cat” and some of which are labeled “Not cat”. During training, feature maps at the top level come to recognize human-intelligible image elements such as edges, while those in deeper levels come to recognize more abstract aspects of images less easily described by the human observers. In these steps, information processing takes place via matrix operations and nonlinearization, while information storage, retention of knowledge acquired during training, takes place as changes in values stored in the filter matrices and weights of inputs to the downstream “neurons”. c) A network that translates. A “simple” NN descended from that in references^, . This network translates between black and white images of cats and otherwise identical images that reveal the cats’ third or inner eyes. During training, pixelated M1 images are subjected to the series of the convolution, nonlinearization, and pooling steps as in panel b. Here, however, entries in downstreammost pooled maps are successively “upsampled” and combined with information coming from intermediate level feature maps on the left. The result for each pixel is compared with pixel intensities for the training M2 images. Training continues by adjustment of filter values until the network learns relationship between M1 and M2 images. The trained network can then operate on a new M1 image to produce a predicted M2 image.

Fig 1.

a) Translating to predict. Top. Network in Panel c is trained on pairs of images of cells acquired under Modality M1 (eg. brightfield) and Modality M2 (here, fluorescence images of signal from a protein the nuclear envelope). Bottom. Given a new brightfield image, the trained network predicts the nuclear envelope signal from information in an image of a field of unlabelled cells. b) A network that classifies. A simple NN similar to Alexnet that identifies images of cats. Its operation shown in captions. In an untrained network, values of the filter matrices (which are convolved with upstream information to create feature maps) and of the weights of the individual inputs to the downstream “fully connected” “neurons” (affine operations and activation functions) are set randomly, and values of the filter matrices and input weights move toward optimum values during training. The network is trained by exposure to images, some of which are labeled “Cat” and some of which are labeled “Not cat”. During training, feature maps at the top level come to recognize human-intelligible image elements such as edges, while those in deeper levels come to recognize more abstract aspects of images less easily described by the human observers. In these steps, information processing takes place via matrix operations and nonlinearization, while information storage, retention of knowledge acquired during training, takes place as changes in values stored in the filter matrices and weights of inputs to the downstream “neurons”. c) A network that translates. A “simple” NN descended from that in references^, . This network translates between black and white images of cats and otherwise identical images that reveal the cats’ third or inner eyes. During training, pixelated M1 images are subjected to the series of the convolution, nonlinearization, and pooling steps as in panel b. Here, however, entries in downstreammost pooled maps are successively “upsampled” and combined with information coming from intermediate level feature maps on the left. The result for each pixel is compared with pixel intensities for the training M2 images. Training continues by adjustment of filter values until the network learns relationship between M1 and M2 images. The trained network can then operate on a new M1 image to produce a predicted M2 image.