Registration of spatio-temporal point clouds of plants for phenotyping

Abstract

Plant phenotyping is a central task in crop science and plant breeding. It involves measuring plant traits to describe the anatomy and physiology of plants and is used for deriving traits and evaluating plant performance. Traditional methods for phenotyping are often time-consuming operations involving substantial manual labor. The availability of 3D sensor data of plants obtained from laser scanners or modern depth cameras offers the potential to automate several of these phenotyping tasks. This automation can scale up the phenotyping measurements and evaluations that have to be performed to a larger number of plant samples and at a finer spatial and temporal resolution. In this paper, we investigate the problem of registering 3D point clouds of the plants over time and space. This means that we determine correspondences between point clouds of plants taken at different points in time and register them using a new, non-rigid registration approach. This approach has the potential to form the backbone for phenotyping applications aimed at tracking the traits of plants over time. The registration task involves finding data associations between measurements taken at different times while the plants grow and change their appearance, allowing 3D models taken at different points in time to be compared with each other. Registering plants over time is challenging due to its anisotropic growth, changing topology, and non-rigid motion in between the time of the measurements. Thus, we propose a novel approach that first extracts a compact representation of the plant in the form of a skeleton that encodes both topology and semantic information, and then use this skeletal structure to determine correspondences over time and drive the registration process. Through this approach, we can tackle the data association problem for the time-series point cloud data of plants effectively. We tested our approach on different datasets acquired over time and successfully registered the 3D plant point clouds recorded with a laser scanner. We demonstrate that our method allows for developing systems for automated temporal plant-trait analysis by tracking plant traits at an organ level.

Fig 1. A time-series of 3D point clouds of two plants (maize (top) and tomato (bottom)) captured during its growth.

Our goal is to develop techniques for automatically registering such 3D scans captured under challenging conditions of changing topology and anisotropic growth of the plant.

Fig 2. Extracting skeletal structure for using semantics of the plant.

The figure illustrates the skeletonization pipeline for a maize (top) and tomato (bottom) plant scan. Note that for the tomato plant, we classify individual leaflets (green + yellow + light-blue) as separate instances rather than as an individual leaf. The leaflets can be combined into a single leaf in case this distinction is not desired/required for the application.

Fig 3

Left: Skeletal matching for an example pair of plant point clouds with the variables involved. Right: Hidden Markov model (HMM) used for correspondence estimation. We only show a subset of the hidden variables, i.e. the potential correspondences, in the HMM. The red line depicts the sequence of best correspondence estimated by the Viterbi algorithm. This produces the correspondences between ${\mathcal{S}}_1$ and ${\mathcal{S}}_2$ visualized with the dash-lined arrows on the left.

Fig 4

Left: Registering the skeleton pair involves estimating the deformation parameters attached to the nodes of the source skeleton ${\mathcal{S}}_1$. Right: Transferring the deformation results to the entire point cloud.

Fig 5. Semantic classification of maize (top) and tomato (bottom) point clouds.

Each stem and leaf (or leaflet) instance is visualized with a different color. Note that the colors of same leaf instances do not correspond over time, as data associations have not been computed at this stage.

Fig 6. Example ground truth labels used in the evaluation of semantic classification of maize (left) and tomato (right) point clouds.

The instance wise labels for each plant organ have been manually annotated by a human user.

Fig 7. 4D registration of a point cloud pair scanned on consecutive days for maize (top) and tomato (bottom) plant.

The left column shows the two input point clouds (${\mathcal{P}}_1,{\mathcal{P}}_2$) along with their skeletons, with the estimated correspondences between the skeleton nodes shown by dashed lines, and the right column shows the deformed point cloud ${\widehat{\mathcal{P}}}_1$ (in pink) overlaid on ${\mathcal{P}}_2$.

Fig 8. Visualizing registration error.

We visualize the registration error as a heatmap for two pairs of tomato plant scans, Day 1 vs. Day 2 and Day 6 vs. Day 10. Blue represents low registration error whereas yellow represents a larger error.

Fig 9. Tracking phenotypic traits for individual organs of the plant.

Our registration procedures allows us to track the growth of the stem and different leave lengths over time and detect topological events such as the emergence of new leaves. Different shades of blue and green in these plots represent individual leaf instances in the first two columns. The orange and red represent the length and diameter of the stem respectively.

Fig 10. Interpolation of point clouds at intermediate time intervals.

Point clouds (gray) at time t − 1 and t come from actual scan measurements whereas the points clouds (pink) at time instants $t_1^i,t_2^i,t_3^i$ visualize the three interpolated scans.