Self-driving microscopy detects the onset of protein aggregation and enables intelligent Brillouin imaging

Abstract

The process of protein aggregation, central to neurodegenerative diseases like Huntington's, is challenging to study due to its unpredictable nature and relatively rapid kinetics. Understanding its biomechanics is crucial for unraveling its role in disease progression and cellular toxicity. Brillouin microscopy offers unique advantages for studying biomechanical properties, yet is limited by slow imaging speed, complicating its use for rapid and dynamic processes like protein aggregation. To overcome these limitations, we developed a self-driving microscope that uses deep learning to predict the onset of aggregation from a single fluorescence image of soluble protein, achieving 91% accuracy. The system triggers optimized multimodal imaging when aggregation is imminent, enabling intelligent Brillouin microscopy of this dynamic biomechanical process. Furthermore, we demonstrate that by detecting mature aggregates in real time using brightfield images and a neural network, Brillouin microscopy can be used to study their biomechanical properties without the need for fluorescence labeling, minimizing phototoxicity and preserving sample health. This autonomous microscopy approach advances the study of aggregation kinetics and biomechanics in living cells, offering a powerful tool for investigating the role of protein misfolding and aggregation in neurodegeneration.

Fig. 1. Self-driving microscopy predicts the onset of protein aggregation.

a The process of protein aggregation occurs unpredictably in different locations at different times, over a large time window of 48 h, yet the transition from soluble protein to mature aggregate is stochastic and relatively rapid (few minutes–few tens of minutes), making it difficult to capture optimally. Images are representative of experiments conducted more than three independent times. Scale bar: 200 µm. b Self-driving microscopy is an approach that uses a feedback loop to autonomously adjust the imaging settings once an event is detected by a real-time model, offering advantages such as optimal use of time and computing resources. c In standard microscopy, images are acquired and then later processed and analyzed. In our SDM approach, processing and computation are done in real-time by a neural network model, AEGON, that was previously trained. This model is capable of detecting the onset of protein aggregation, triggering an optimized acquisition to ideally capture the process. Images are representative of experiments conducted more than three independent times. Scale bars: 5 µm.

Fig. 2. Optimized multimodal imaging triggered by our model, AEGON.

a Classification matrix of our model AEGON for predicting aggregation onset. Performance metrics: precision = 1.0, recall =  0.82, F1 score = 0.9, accuracy = 0.91. b Test-set examples of correctly predicted events and correctly predicted non-events. Inset images show the eventual fate of these fields of view. Training and test set data were collected from more than three independent experiments. c Multimodal (fluorescence and QPI) images captured after an event is detected by our model within our SDM pipeline. The intensity in each image is locally rescaled between 0 and 1. Images are representative of experiments conducted more than three independent times. Scale bars: 5 µm.

Fig. 3. Generalizability of the method enables dynamic Brillouin microscopy.

a Accuracy of the AEGON models we trained for various time-point inputs. One-time-point models maintain high accuracy. b Accuracy of our one-time-point ViT model with different z-plane inputs. The one-plane model maintains high accuracy. c With our one-plane, one-time-point model, only one fluorescence image is needed to predict protein aggregation. We are able to use our model on a completely different setup and adopt our SDM pipeline, without transfer learning. d Multimodal imaging on a different setup, triggered by our AEGON model, enabling dynamic Brillouin microscopy of the aggregation process. Arrows mark the aggregating region (different colors only serve to optimize the contrast). Images are representative of experiments conducted more than three independent times. Scale bars: 5 µm.

Fig. 4. Evolution and comparison of the Brillouin shift and linewidth over time.

a Dynamic imaging of the linewidths throughout aggregation is enabled by the model, which predicts the onset of the process. Scale bar: 5 µm. b, c Comparison of the Brillouin shifts and linewidths of the aggregating region versus the periphery. TP–timepoint. Data are presented as mean values ± standard deviation, within the respective region in the image, at each timepoint for the shown example. b Brillouin shift comparison. c Linewidth comparison.

Fig. 5. Real-time identification and biomechanical analysis of aggregates using SDM and IC-LINA.

The IC-LINA model detects in real time the presence of aggregates solely from label-free brightfield images, eliminating the need for fluorescence imaging to localize aggregates. Here, the fluorescence image serves as a ground truth to validate aggregate presence. Autonomously-triggered Brillouin microscopy enables detailed biomechanical analysis at identified locations. Images are representative of experiments conducted more than three independent times. Scale bars: 5 µm.