An open-source microscopy framework for simultaneous control of image acquisition, reconstruction, and analysis

Abstract

We present a computational framework to simultaneously perform image acquisition, reconstruction, and analysis in the context of open-source microscopy automation. The setup features multiple computer units intersecting software with hardware devices and achieves automation using python scripts. In practice, script files are executed in the acquisition computer and can perform any experiment by modifying the state of the hardware devices and accessing experimental data. The presented framework achieves concurrency by using multiple instances of ImSwitch and napari working simultaneously. ImSwitch is a flexible and modular open-source software package for microscope control, and napari is a multidimensional image viewer for scientific image analysis. The presented framework implements a system based on file watching, where multiple units monitor a filesystem that acts as the synchronization primitive. The proposed solution is valid for any microscope setup, supporting various biological applications. The only necessary element is a shared filesystem, common in any standard laboratory, even in resource-constrained settings. The file watcher functionality in Python can be easily integrated into other python-based software. We demonstrate the proposed solution by performing tiling experiments using the molecular nanoscale live imaging with sectioning ability (MoNaLISA) microscope, a high-throughput super-resolution microscope based on reversible saturable optical fluorescence transitions (RESOLFT).

Keywords: Automation; RESOLFT; Software.

Graphical abstract

Fig. 1

General framework for simultaneous experiment acquisition, image reconstruction, and visualization using multiple computational units. The microscope experiments are performed in the acquisition unit by executing python scripts, which are created and distributed by the orchestrator unit. The user specifies the acquisition parameters in the scripts, such as number of tiles and laser powers. The recorded data is saved on disk (raw data), which acts as a queue for the image reconstruction unit. The image reconstruction unit turns the acquired raw data into tiles in a parallelized manner (see timeline in the top left corner) and can be visualized and post-processed in the orchestrator unit. Repeating this cycle gives an image of a neuron with an increased FOV.

Fig. 2

Software pipeline using multiple units. a. The orchestrator is an instance of napari with the napari-file-watcher plugin. The user interacts with the orchestrator unit by defining experiments as python scripts, which are then added to the filesystem using the ImSwitch scripting widget. Once the reconstructed images are received, they are displayed as napari layers available for post-processing. b. The logic for each file watcher in Python. It periodically monitors a filesystem (FS), adds the incoming files to a queue and sends them using the sigNewFiles signal. c. The acquisition unit is an instance of ImSwitch. The file watcher widget displays and tracks the new files in each folder. Whenever new files are in the FS, it adds them to a queue and executes them sequentially. After every execution, the signal sigScriptExecutionFinished is called. d. The reconstruction unit is also an instance of ImSwitch, featuring the imreconstruct module. The workflow is similar to the acquisition, with the difference that the files are raw images that are reconstructed with an image processing algorithm. The reconstruction function is not called directly but through the sigReconstruct signal instead. This choice is due to the architecture of the imreconstruct module. Solid lines represent direct connections, and dotted lines are calls using python signals.

Fig. 3

ImSwitch GUI, acquisition unit. The GUI implements different widgets for the control of devices (“Device control widgets”) and experimental acquisition (“Detector and image settings”). The “Data storage and recording” module saves data into disk or memory. The file watcher and other specialized widgets are implemented in “File watcher and optional widgets”.

Fig. 4

ImSwitch GUI, reconstruction unit. The raw data is displayed in the “Raw data loading widgets” section, and the reconstructed data is in the “Reconstructed data visualization” module. The reconstruction algorithm parameters can be set in “Reconstruction algorithm settings”.

Fig. 5

Napari GUI and the napari-file-watcher plugin, orchestrator unit. The plugin contains two widgets: “ImSwitch Scripting” for editing and creating execution scripts and the “File Watcher” waits for new images to be displayed.

Fig. 6

Timelapse ImSwitch automation script. N = 10 lapses are executed in the (x,y,z) stage position coordinates. The ImSwitch API is exported and accessible with the “api.imcontrol” call.

Fig. 7

ImSwitch GUI in acquisition unit, imscripting module for editing and executing scripts. The scripts can be loaded in “Filesystem browser”, edited, and executed in the “Script editing” section, and a “Console” is implemented for debugging purposes.

Fig. 8

Cyclic time-lapse imaging of mitochondria in U2OS cells in a 2x2 (A, B, C, D) tile array. a The acquisition script (timelapse-tiling.py) drives the microscopy automation acquisition by calling ImSwitch functions. Concurrent processing enables subsequent acquisition and reconstruction of the image tiles. b Cyclic timelapse images are performed by imaging each of the tiles and sequentially repeating the procedure. Scale bar 5 µm. c-d The super-resolution features can be observed during a prolonged time (10 super-resolved frames) for each tile independently in U2OS cells expressing OMP25-rsEGFP2. A zoom-in visualizes the dynamics of selected areas in both confocal and MoNaLISA modalities. Scale bars 0.5 µm and 5 µm, respectively.

Fig. 9

Selective tiling for extending the FOV to 160x160µm² (25x25 tiles). a The focus positions are registered by the user using a widefield laser and an ImSwitch widget. The user can skip the tile, thus adapting to the image content. The microscope then performs selective tiling by imaging only the selected areas. b-c Schematic of the tiling scheme, including 14 % overlap, and visualization of a single tile (38x38µm²). d-e Imaging the actin cytoskeleton of U2OS cells and primary hippocampal neurons expressing actinChromobody-rsEGFP2. Scale bar 20 µm. f-h zooming in Tile 1 in the U2OS cells (d), and comparison between confocal and MoNaLISA. Scale bars 5 µm and 0.5 µm, respectively. i Reconstruction time of 200 executions using a remote instance. Each color represents a different experiment, and each data point is a single image acquisition (or tile). The values were extracted from the logger files for all the experiments performed using the framework (200 acquisitions).