Labeling lateral prefrontal sulci using spherical data augmentation and context-aware training

Abstract

The inference of cortical sulcal labels often focuses on deep (primary and secondary) sulcal regions, whereas shallow (tertiary) sulcal regions are largely overlooked in the literature due to the scarcity of manual/well-defined annotations and their large neuroanatomical variability. In this paper, we present an automated framework for regional labeling of both primary/secondary and tertiary sulci of the dorsal portion of lateral prefrontal cortex (LPFC) using spherical convolutional neural networks. We propose two core components that enhance the inference of sulcal labels to overcome such large neuroanatomical variability: (1) surface data augmentation and (2) context-aware training. (1) To take into account neuroanatomical variability, we synthesize training data from the proposed feature space that embeds intermediate deformation trajectories of spherical data in a rigid to non-rigid fashion, which bridges an augmentation gap in conventional rotation data augmentation. (2) Moreover, we design a two-stage training process to improve labeling accuracy of tertiary sulci by informing the biological associations in neuroanatomy: inference of primary/secondary sulci and then their spatial likelihood to guide the definition of tertiary sulci. In the experiments, we evaluate our method on 13 deep and shallow sulci of human LPFC in two independent data sets with different age ranges: pediatric (N=60) and adult (N=36) cohorts. We compare the proposed method with a conventional multi-atlas approach and spherical convolutional neural networks without/with rotation data augmentation. In both cohorts, the proposed data augmentation improves labeling accuracy of deep and shallow sulci over the baselines, and the proposed context-aware training offers further improvement in the labeling of shallow sulci over the proposed data augmentation. We share our tools with the field and discuss applications of our results for understanding neuroanatomical-functional organization of LPFC and the rest of cortex (https://github.com/ilwoolyu/SphericalLabeling).

Keywords: Context encoder; Cortical surface; Frontal cortex; Spherical data augmentation; Sulcal labeling.

Fig. 1.

Primary, secondary, and tertiary sulci in LPFC emerge at different timepoints in gestation and are related to morphological variability in surface area and sulcal depth. (a) white matter (top) and inflated cortical surfaces (bottom) from five individual subjects zoomed in on LPFC in the left hemisphere. Light gray vertices are sulci, dark gray vertices are gyri, and colors indicate manually identified primary/secondary and tertiary sulci. Sulci in LPFC, especially tertiary sulci, have high spatial variability as well as various branches that differ across individuals as well as between hemispheres in the same individual. This variability makes it challenging to manually label sulci in LPFC, as well as generate automated tools that accurately define LPFC tertiary sulci. (b) sulci in LPFC can be clustered into different categories based on the point in which they emerge in gestation: primary sulci emerge first, while tertiary sulci emerge last and secondary sulci emerge in between. Interestingly, these gestational timepoint differences are also related to morphological differences: primary sulci are largest (in terms of surface area) and deepest, while tertiary sulci are typically smallest, and also shallowest, while secondary sulci are in between. We consider two groups in the present work: primary/secondary sulci (blue) and tertiary sulci (red) as determined by modern and classic neuroanatomy studies (Armstrong et al., 1995; Chi et al., 1977; Connolly, 1950; Cunningham, 1892; Miller et al., 2020b; 2020c; Retzius, 1896; Sanides, 1962; 1964; Weiner, 2019; Weiner et al., 2014; Weiner and Zilles, 2016; Welker, 1990). Each data point indicates total surface area and average sulcal depth per sulcus in LPFC. A total of 13 sulci from 96 subjects are used for this visualization. See Table 1 for details of the labeling protocol and Section 3.1 for the data collection.

Fig. 2.

A schematic overview of the proposed method. Our method consists of two main components in the learning phase: surface data augmentation (blue box) and context-aware training (green box). During data augmentation, we augment training samples (dotted box) by deforming surface data over the sphere via surface registration to every possible pair of training samples. We decompose spherical deformation via the spherical harmonics and reconstruct its intermediate deformation by controlling the basis functions. In this way, a gap can be filled between moving and target samples in the feature space along their deformation trajectory (red). In the context-aware training, spatial information of primary/secondary sulci are inferred to guide labels of tertiary sulci. The information except for the background label (gray node in the output likelihoods) is then fed into the second training stage to offer spatial clues to guide the labeling of tertiary sulci. Note that the sulcal labeling in the test phase does not use registered (deformed) surface data, which allows fast annotation.

Fig. 3.

The proposed data augmentation. Top/Middle: input (binarized) spherical data (geometric feature) F is deformed to that of a target sphere. After rigid rotation (F⁰), the deformed spherical data become closer as a degree of spherical harmonics increases. Bottom: the manual annotation Z is driven by the intermediate deformation. Even with the improved geometric feature matching, manual annotation does not necessarily match due to spatial inconsistency and various sulcal branches (yellow box). This implies that a single training sample is insufficient to capture variability of sulci in LPFC. For data augmentation, the proposed method utilizes intermediate deformation of the combinatorial registration, by which sulcal variability can be better captured than a single training sample. Thus, model training is generalized by learning neuroanatomical variations provided by manual annotation for a set of similar spherical data (geometric features), to which the enhanced inference of unseen data belongs.

Fig. 4.

Improved spatial coherence. (a) cortical sulci tend to have a relatively small triangle size in the pial surface. (b) the maximum likelihood-based inference often yields isolated regions (yellow). (c) a standard graph-cut technique is used to refine spatial coherence. From the observation of adaptive triangle size in the pial surface, neighborhood relationship is encoded by the edge length. (d) the resulting labels become more spatially coherent to the manual annotation.

Fig. 5.

Visual inspection of label inference on example subjects around the average performance (top: pediatric sample, bottom: adult sample). The multi-atlas approach (second column from left) lacks geometric details of labeled regions along cortical folds as indicated by the rather low dice coefficient compared to the rightmost column. The conventional training without data augmentation (third column from left: Naive) shows poor performance that generates small isolated segments even after improving spatial coherence due to limited generalizability particularly with a small sample size. Although rotation data augmentation (fourth column from left) offers higher accuracy than the conventional training, the inference is more improved (generalized) with the proposed data augmentation, and the labeling accuracy of tertiary sulci can be further improved by introducing context information (two rightmost columns: Non-rigid and Non-rigid+Context). Note that there is even considerable variability in sulci that emerge relatively early in gestation such as sprs and iprs. In the pediatric example (top left), the reader can appreciate the annectant gyral components (plis de passage; Gratiolet (1854); Mangin et al. (2019); Parent (2014); yellow box) between both iprs and sprs.

Fig. 6.

Dice overlap per sulcus in the pediatric cohort in LPFC. The statistical significance is reported after multi-comparison correction among the 13 sulci (FDR at q= .05). The proposed data augmentation (Non-rigid) shows higher accuracy (Dice overlap) than the baseline methods. After the context-aware training, the Dice overlap is further improved in primary/secondary (cs (left hemisphere); iprs and sfs_p (right hemisphere)) and tertiary (pmfs_a, imfs_v, and mfms (left hemisphere); ifms (right hemisphere)) sulci compared to the conventional training with rotation data augmentation. Importantly, the proposed method does not perform worse than the baseline methods for any sulcus. Legend: standard errors (hat); significant improvement compared to the baseline methods for Non-rigid+Context* (blue); for Non-rigid or Non-rigid+Context* (black).

Fig. 7.

Dice overlap per sulcus in the adult cohort in LPFC. The statistical significance is reported after multi-comparison correction among the 13 sulci (FDR at q= .05). The proposed data augmentation shows higher accuracy (Dice overlap) than the baseline methods. After the context-aware training, the Dice overlap is further improved in primary/secondary (sprs (right hemisphere)) and tertiary (pmfs_a, imfs_h, and imfs_v (left hemisphere); imfs_v and ifms (right hemisphere)) sulci compared to the conventional training with rotation data augmentation. Importantly, the proposed method does not perform worse compared to the baseline methods for any sulcus. Legend: standard errors (hat); significant improvement than the baseline methods for Non-rigid+Context* (blue); for Non-rigid or Non-rigid+Context* (black).