A survey of the impact of self-supervised pretraining for diagnostic tasks in medical X-ray, CT, MRI, and ultrasound

Abstract

Self-supervised pretraining has been observed to be effective at improving feature representations for transfer learning, leveraging large amounts of unlabelled data. This review summarizes recent research into its usage in X-ray, computed tomography, magnetic resonance, and ultrasound imaging, concentrating on studies that compare self-supervised pretraining to fully supervised learning for diagnostic tasks such as classification and segmentation. The most pertinent finding is that self-supervised pretraining generally improves downstream task performance compared to full supervision, most prominently when unlabelled examples greatly outnumber labelled examples. Based on the aggregate evidence, recommendations are provided for practitioners considering using self-supervised learning. Motivated by limitations identified in current research, directions and practices for future study are suggested, such as integrating clinical knowledge with theoretically justified self-supervised learning methods, evaluating on public datasets, growing the modest body of evidence for ultrasound, and characterizing the impact of self-supervised pretraining on generalization.

Keywords: Computed tomography; Machine learning; Magnetic resonance imaging; Radiology; Representation learning; Self-supervised learning; Ultrasound; X-ray.

Fig. 1

Example of a typical SSL workflow, with an application to chest X-ray classification. (1) Self-supervised pretraining: A parameterized model $g_{\varphi }(f_{\theta }(\mathbf{x}))$ is trained to solve a pretext task using only the chest X-rays. The labels for the pretext task are determined from the inputs themselves, and the model is trained to minimize the pretext objective ${\mathcal{L}}_{\text{pre}}$. At the end of this step, $f_{\theta }$ should output useful feature representations. (2) Supervised fine-tuning: Parameterized model $q_{\psi }(f_{\theta }(\mathbf{x}))$ is trained to solve the supervised learning task of chest X-ray classification using labels specific to the classification task. Note that the previously learned $f_{\theta }$ is reused for this task, as it produces feature representations specific to chest X-rays

Fig. 2

Breakdown of the papers included in this survey by a imaging modality and b year of publication

Fig. 3

Examples of generative SSL pretext tasks

Fig. 4

Examples of predictive SSL pretext tasks

Fig. 5

A depiction of the forward pass for a positive pair in a standard noncontrastive pretext task. An image is subject to stochastic data transformations twice, producing distorted views ${\mathbf{x}}_a$ and ${\mathbf{x}}_b$, which are passed through the feature extractor $f_{\theta }$ to yield feature representations ${\mathbf{h}}_a$ and ${\mathbf{h}}_b$. The projector $g_{\varphi }$ transforms ${\mathbf{h}}_a$ and ${\mathbf{h}}_b$ into embeddings ${\mathbf{z}}_a$ and ${\mathbf{z}}_b$ respectively. Typically, the objective $\mathcal{L}$ is optimized to maximize the similarity of ${\mathbf{z}}_a$ and ${\mathbf{z}}_b$