GPU-accelerated real-time reconstruction in Python of three-dimensional datasets from structured illumination microscopy with hexagonal patterns

Abstract

We present a structured illumination microscopy system that projects a hexagonal pattern by the interference among three coherent beams, suitable for implementation in a light-sheet geometry. Seven images acquired as the illumination pattern is shifted laterally can be processed to produce a super-resolved image that surpasses the diffraction-limited resolution by a factor of over 2 in an exemplar light-sheet arrangement. Three methods of processing data are discussed depending on whether the raw images are available in groups of seven, individually in a stream or as a larger batch representing a three-dimensional stack. We show that imaging axially moving samples can introduce artefacts, visible as fine structures in the processed images. However, these artefacts are easily removed by a filtering operation carried out as part of the batch processing algorithm for three-dimensional stacks. The reconstruction algorithms implemented in Python include specific optimizations for calculation on a graphics processing unit and we demonstrate its operation on experimental data of static objects and on simulated data of moving objects. We show that the software can process over 239 input raw frames per second at 512 × 512 pixels, generating over 34 super-resolved frames per second at 1024 × 1024 pixels. This article is part of the Theo Murphy meeting issue 'Super-resolution structured illumination microscopy (part 1)'.

Keywords: GPU processing; fluorescence microscopy; light-sheet microscopy; reconstruction algorithm; structured illumination microscopy; super-resolution microscopy.

Figure 1.

The design of a light sheet SIM system with hexagonal patterned illumination. Three light sheets (green) are coincident at mutual angles of 120° in the focal plane of the microscope to form a hexagonal structured light sheet. Cells flowing through the light sheet in the +z-direction emit fluorescence that is collected in the +z-direction by an objective lens and imaged onto a camera. (Online version in colour.)

Figure 2.

Illumination pattern design for hexSIM. (a) The locations of the three illumination beams (red stars) and the seven spatial frequencies from the illumination pattern (blue spots) relative to the illumination pupil (grey circle). (b) Simulated illumination pattern for hexSIM. (c) Potential light-sheet-based illumination scheme for hexSIM. (Online version in colour.)

Figure 3.

The flowchart of hexSIM analysis process. The calibration phase measures the parameters of the applied structured illumination from a block of seven acquired raw input images of the sample. Along with other user supplied reconstruction parameters, it then calculates constant arrays that are used in the reconstruction phase. The reconstruction phase consists of three stages as shown taking the input data, pre-filtering and up-sampling it, unmixing and remixing the different carrier bands in the image and finally post-filtering to produce an output super-resolved image. (Online version in colour.)

Figure 4.

The strategies of different reconstruction process: standard reconstruction, frame-by-frame reconstruction and batch reconstruction. (Online version in colour.)

Figure 5.

Results of processing a stack of 280 images of a simulated moving object (a) projection along the x-axis of the super-resolved stack image showing reconstruction artefacts (c) as a result of the object motion. (b) Slice in yz plane of the Fourier spectrum of the 3D image shown in (a). (d) Slice in yz plane of the Fourier spectrum of the reconstructed 3D image shown in (c). (f) Spectrum in (d) filtered to just the zero band in z and the spatial domain reconstruction of this showing reduced artefacts (e). (g,h) The results of the batch reconstruction method that perform the filtering in z as part of the unmixing–remixing process and results in a 7x reduction in data to just 40 planes and a close to Nyquist sampled output image. Videos of the data represented in (a,c,e,g) as the xy-plane is moved through z are given in the electronic supplementary material, Movies S1–S4, respectively. The data shown in (a) are also filtered in z in the same way as that shown in (e) in order to generate a wide-field z-stack and are shown in electronic supplementary material, Movie S5. (Online version in colour.)

Figure 6.

(a–f) Calibration of the hexSIM system with a fluorescent microspheres sample (0.1 µm TetraSpeck™ microspheres, ThermoFisher). Wide-field image (a), hexSIM image (b) and fairSIM image (c). From a decorrelation analysis, the estimated resolution of the wide-field image is 474 nm. For the hexSIM image and fairSIM, it is 267 nm and 259 nm, respectively. (d–f) Experimental results of imaging the BPAE cells (FluoCells Prepared Slide #1, ThermoFisher). Comparison of the wild-field image (d), hexSIM image (e) and fairSIM image (f). The estimated resolutions are 528, 296 and 266 nm from left to right. Zoomed in views of the same area are provided in the down left corner. The hexSIM reconstructed the image from a dataset of seven raw images. The fairSIM processed image is reconstructed from a linear decomposition of this dataset into nine images. (Online version in colour.)

Figure 7.

A 3D iso-surface reconstruction of a 1000 point cloud dataset distributed randomly over a 10 µm diameter sphere. (a) A conventional wide-field image reconstructed from the simulated data while (b) shows the super-resolved image reconstruction. A movie file of these data is available in the electronic supplementary material, Movie S6. (Online version in colour.)