A deep learning algorithm for 3D cell detection in whole mouse brain image datasets

Abstract

Understanding the function of the nervous system necessitates mapping the spatial distributions of its constituent cells defined by function, anatomy or gene expression. Recently, developments in tissue preparation and microscopy allow cellular populations to be imaged throughout the entire rodent brain. However, mapping these neurons manually is prone to bias and is often impractically time consuming. Here we present an open-source algorithm for fully automated 3D detection of neuronal somata in mouse whole-brain microscopy images using standard desktop computer hardware. We demonstrate the applicability and power of our approach by mapping the brain-wide locations of large populations of cells labeled with cytoplasmic fluorescent proteins expressed via retrograde trans-synaptic viral infection.

Fig 1. Simplified schematic diagram of the serial two-photon microscope and data acquisition process.

A: The tissue is excited using a femtosecond Ti-sapphire laser (emission wavelength = 800 nm). For data collection, 50 μm of tissue (at approximately 40 μm to 90 μm below the tissue surface) is imaged in ten, 5 μm thick planes. An in-built microtome then physically removes a 50 μm thick section from the optical face. This process is repeated to generate a complete 3D dataset of the specimen. B: For signal collection, the emitted lightpath is split into two channels whereby the primary channel detects the fluorescence signal of interest from labelled cells (e.g. mCherry at 610 nm) and the second channel (e.g. at 450 nm) detects the tissue autofluorescence signal that reveals gross anatomical structure.

Fig 2. Illustration of the cell detection process.

A: Single coronal plane of raw data (primary signal channel, Ch1). B: Enlarged insert of cortical region from A showing examples of structural features (artefacts) often erroneously detected. C: Cortical region shown in B, in the secondary autofluorescence channel (Ch2). Cells can only be seen in Ch1, but artefacts are visible in both channels. D: Detected cell candidates overlaid on raw data. Labelled cells as well as numerous artefacts are detected. E: Illustration of training data. A subset of detected cell candidates are classified as cells (yellow) or artefacts (purple). Cuboids of data centered on these selected cell candidates are then used to train the network. F: Classified cell candidates. The trained cell classification network is applied to all the cell candidates from (E) and correctly rejects the initial false positives.

Fig 3. Cell classification.

A: The input data to the modified ResNet are 3D image cuboids (50 μm x 50 μm x 100 μm) centered on each cell candidate. There are two cuboids, one from the primary signal channel, and one from the secondary autofluorescence channel. The data is then fed through the network, resulting in a binary label of cell or non-cell. During training the network “learns” to distinguish true cells, from other bright non-cellular objects. See S3 and S4 Figs for more details of the 3D ResNet architecture. B: Training the initial cell classification network: classification accuracy as a function of training data quantity.

Fig 4. Application of the algorithm to unseen data. Presynaptic neurons labelled by rabies viral injection into layer 2/3 primary visual cortex in Penk-Cre mice.

A: Detected cells overlaid on raw data along with the brain region segmentation for Brain 1 (blue) and 2 (orange). The size of the coloured disk represents the proximity of the cell centroid to the image plane displayed. Brain regions with detected cells shown: RSP—Retrosplenial area, VISp—Primary visual area, VISl—lateral visual area, VISli—Laterointermediate area, TEa—Temporal association areas. B: Comparison of cell detection before and after re-training the pre-trained network in different regions of the image. Cells with different morphologies are correctly detected in both dense and sparse regions, and artefacts are rejected. VISli—Laterointermediate area, LGd—Dorsal part of the lateral geniculate complex, DR—Dorsal nucleus raphe, SC/M—Superior colliculus & meninges. C: Comparison of cell counts per ARA brain region between the algorithm and the mean of the two expert counts. Lighter points represent results before re-training and darker points after re-training. Best fit shown after re-training. Five regions with the highest number of detected cells coloured, VISp2/3—Primary visual area, layer 2/3, VISp5—Primary visual area, layer 5, LGd-co—Dorsal part of the lateral geniculate complex, core, LP—Lateral posterior nucleus of the thalamus, VISp4—Primary visual area, layer 4. D: Visualisation of detected cells from both brains warped to the ARA coordinate space in 3D, along with the rabies virus injection site target (primary visual cortex, wireframe).