An Open-Source Package for Thermal and Multispectral Image Analysis for Plants in Glasshouse

Abstract

Advanced plant phenotyping techniques to measure biophysical traits of crops are helping to deliver improved crop varieties faster. Phenotyping of plants using different sensors for image acquisition and its analysis with novel computational algorithms are increasingly being adapted to measure plant traits. Thermal and multispectral imagery provides novel opportunities to reliably phenotype crop genotypes tested for biotic and abiotic stresses under glasshouse conditions. However, optimization for image acquisition, pre-processing, and analysis is required to correct for optical distortion, image co-registration, radiometric rescaling, and illumination correction. This study provides a computational pipeline that optimizes these issues and synchronizes image acquisition from thermal and multispectral sensors. The image processing pipeline provides a processed stacked image comprising RGB, green, red, NIR, red edge, and thermal, containing only the pixels present in the object of interest, e.g., plant canopy. These multimodal outputs in thermal and multispectral imageries of the plants can be compared and analysed mutually to provide complementary insights and develop vegetative indices effectively. This study offers digital platform and analytics to monitor early symptoms of biotic and abiotic stresses and to screen a large number of genotypes for improved growth and productivity. The pipeline is packaged as open source and is hosted online so that it can be utilized by researchers working with similar sensors for crop phenotyping.

Keywords: co-registration; illumination correction; image processing; multispectral; segmentation; thermal.

Figure 1

Integrated multimodal and imaging setup. A multispectral camera was physically mounted on the top of a thermal camera using a magnetic mount assembly to provide a uniform field-of-view (FoV). The irradiance sensor was mounted on the top. During imaging, a radiometric calibration target with 80 percent reflectivity was placed behind the plants.

Figure 2

The image processing pipeline. The steps include image acquisition by thermal and multispectral cameras, and image processing for distortion correction of multispectral images, image registration (coarse and fine), radiometric scaling of thermal images, and illumination correction of multispectral images.

Figure 3

Process of capturing images and radial optical distortion. (a) An image without distortion (top) and image with radial barrel distortion (bottom) where r is the off-axis image distance, which increases with distortion. (b) Extrinsic parameters (Rotation (R) and Translation (T)) are used to convert the 3D world plane coordinates (${\mathrm{O}}_{\mathrm{w}}$) to a 3D camera plane coordinates (${\mathrm{O}}_{\mathrm{c}}$), which are converted to 2D image coordinates (${\mathrm{O}}_{\mathrm{i}}$) with the help of intrinsic parameters; ${\mathrm{O}}_{\mathrm{p}}$ represents the pixel plane.

Figure 4

(a) A distorted image of a checkerboard pattern from a multispectral camera and (b) extrinsic parameters action visualisation. Images were taken from different angles and distances to calculate extrinsic parameters and minimise radial barrel distortion.

Figure 5

Setup for coarse image registration: (a) FoV of RGB; (b) FoV of thermal image. A white corflute with different geometric cut-outs was placed in front of a black background with a higher surface temperature than the corflute sheet. The cut-outs are visible in optical (RGB), multispectral, and thermal image bands.

Figure 6

(a) Feature detection between RGB and thermal images; the green cross and red circles represent common features (corners) for thermal and multispectral images, respectively. (b) Working principle of the projection of moving image into the FoV of a fixed image using a geometric transformations matrix. The red circle, blue cross, and green cross represent the features of the fixed image, moving image, and projected image, respectively.

Figure 7

Image registration output after (a) coarse registration and (b) fine registration. Pink represents misalignment between the thermal and optical images, which was significantly reduced after fine registration.

Figure 8

Pixel values of images: (a) before radiometric rescaling—the pixel values are stored as Digital Numbers (DNs); (b) after radiometric rescaling—the DNs are converted to temperature values. The maximum and minimum temperature values are recorded on the right of the thermal image.

Figure 9

Output of RGB images after (a) gradient correction and (b) illumination correction.

Figure 10

Image segmentation: (a) foreground mask after adaptive thresholding and (b) segmented RGB image after application of the foreground mask. The non-canopy pixel values are converted to zero.

Figure 11

An eight-band stacked image representing only the canopy pixels in the following order: RGB, green, red, NIR, red-edge, and thermal. Each pixel of an image represents the same pixels for the other images, which are represented by red squares in the images.

Figure 12

Vegetative indices: (a) Normalized Difference VI (NDVI) and (b) Chlorophyll Index red edge (CIre).

Figure 13

Reprojection error between the distorted and undistorted image for NIR band. The x- and y-axis represent the number of images and mean errors in pixels, respectively.

Figure 14

Mattes mutual information between multispectral and thermal images after registration.

Figure 15

Error metric for segmentation: (a) Root Mean Square Error (RMSE) and (b) Structural Similarity Map (SSIM) between the original image and segmented images.