Brain-like Flexible Visual Inference by Harnessing Feedback-Feedforward Alignment

Abstract

In natural vision, feedback connections support versatile visual inference capabilities such as making sense of the occluded or noisy bottom-up sensory information or mediating pure top-down processes such as imagination. However, the mechanisms by which the feedback pathway learns to give rise to these capabilities flexibly are not clear. We propose that top-down effects emerge through alignment between feedforward and feedback pathways, each optimizing its own objectives. To achieve this co-optimization, we introduce Feedback-Feedforward Alignment (FFA), a learning algorithm that leverages feedback and feedforward pathways as mutual credit assignment computational graphs, enabling alignment. In our study, we demonstrate the effectiveness of FFA in co-optimizing classification and reconstruction tasks on widely used MNIST and CIFAR10 datasets. Notably, the alignment mechanism in FFA endows feedback connections with emergent visual inference functions, including denoising, resolving occlusions, hallucination, and imagination. Moreover, FFA offers bio-plausibility compared to traditional back-propagation (BP) methods in implementation. By repurposing the computational graph of credit assignment into a goal-driven feedback pathway, FFA alleviates weight transport problems encountered in BP, enhancing the bio-plausibility of the learning algorithm. Our study presents FFA as a promising proof-of-concept for the mechanisms underlying how feedback connections in the visual cortex support flexible visual functions. This work also contributes to the broader field of visual inference underlying perceptual phenomena and has implications for developing more biologically inspired learning algorithms.

Figure 1:

Feedback-Feedforward Alignment. Learning: backpropagation and feedback alignment train a discriminator with symmetric $W_f^T$ or fixed random R_i weights, respectively. FFA maps input x to latents y as in a discriminator but also reconstructs the input $\widehat{x}$ from the latent. The forward and backward pathways also pass gradients back for their counterpart performing inference in the opposite direction. Inference: We run forward and feedback connections trained under FFA in a loop to update the activations (x) for each of the inference tasks e.g. mental imagery. Δ shows the difference between the input signal and the reconstructed (output). * shows the imposed upper bound. See algorithm 2 in Section 8.4.

Figure 2:

Co-optimization in FFA. A) Accuracy and reconstruction performance for FFA and control algorithms as a function of epochs. B) Dual-task performance for a variety of feedforward discriminative and autoencoder architectures trained under BP or FA compared to FFA training (details for architecture in Suppl. 8.1). The shaded area represents the desired corner. C) Robustness to input Gaussian noise (μ = 0 and varying σ² between 0 and 1) as measured by test accuracy on the noisy input.

Figure 3:

Denoising in FFA. Closed-loop inference on noisy inputs (σ² = 0.4) performed by FFA and control algorithms assuming a static read-out for discrimination set by iteration 0. Shown at right, the sample reconstructions recovered by FFA and control autoencoders over 4 iterations (no clipping or other processing was performed on these images).

Figure 4:

Resolving occlusion. A 15×15 black square occludes the digits in the first columns as shown in the second column. Each row shows a sample occluded digit (5, 8, and 9) and the corresponding resolved images under high noise and low noise conditions. For high noise and low noise visual inference, the resolved digit is depicted in 5th and the last columns, respectively. For details regarding the sample intermediate iterations refer to Figure Suppl. 10.

Figure 5:

Hallucination. Without external input, we let the inference algorithm run on the FFA-trained network until convergence (the last column) for high noise (upper) and low noise (lower) inference. The sample iterations are linearly spaced and for high noise, there are typically twice as many iterations needed. Refer to Section 8.8 for iteration values.

Figure 6:

Visual imagery. Generated samples (upper panels) using the inference algorithm on the FFA-trained network when top-down signal ‘5’ (left) and ‘3’ (right) was activated. The sample iterations (equally spaced) for sample generations were shown in the lower panel. Each row corresponds to an inference noise level. Refer to Section 8.7 for iteration and β values