MInet: A Novel Network Model for Point Cloud Processing by Integrating Multi-Modal Information

Abstract

Three-dimensional LiDAR systems that capture point cloud data enable the simultaneous acquisition of spatial geometry and multi-wavelength intensity information, thereby paving the way for three-dimensional point cloud recognition and processing. However, due to the irregular distribution, low resolution of point clouds, and limited spatial recognition accuracy in complex environments, inherent errors occur in classifying and segmenting the acquired target information. Conversely, two-dimensional visible light images provide real-color information, enabling the distinction of object contours and fine details, thus yielding clear, high-resolution images when desired. The integration of two-dimensional information with point clouds offers complementary advantages. In this paper, we present the incorporation of two-dimensional information to form a multi-modal representation. From this, we extract local features to establish three-dimensional geometric relationships and two-dimensional color relationships. We introduce a novel network model, termed MInet (Multi-Information net), which effectively captures features relating to both two-dimensional color and three-dimensional pose information. This enhanced network model improves feature saliency, thereby facilitating superior segmentation and recognition tasks. We evaluate our MInet architecture using the ShapeNet and ThreeDMatch datasets for point cloud segmentation, and the Stanford dataset for object recognition. The robust results, coupled with quantitative and qualitative experiments, demonstrate the superior performance of our proposed method in point cloud segmentation and object recognition tasks.

Keywords: LiDAR; multi-modal information; object recognition; point cloud; segmentation.

Figure 1

Three-dimensional feature extraction diagram. The SG module is FPS sample and grouping. (For three different scale regions of the same central point, each of the three sets of MLP, the convolution kernel is (32, 64, 128), (64, 128, 256), and (64, 128, 256), and then the three-dimensional features are obtained by concat operation).

Figure 2

Local feature extraction (The local features are obtained by extracting features from three different scales).

Figure 3

Two-dimensional feature extraction diagram (Three different convolution processes were carried out on the input, and the convolution kernels were 16, 32, and 64, respectively. After concat operation was carried out on the results obtained from the convolution processing, three MLP layers with convolution kernels 64, 32, num_class were used to extract two-dimensional information features and obtain the final two-dimensional features).

Figure 4

Object feature extraction (The multivariate features of the target are obtained through concat and MLP layers).

Figure 5

Visualization of segmentation results based on ShapeNet.

Figure 6

Visualization of segmentation results based on 3Dmatch.

Figure 7

Visualization of different methods on S3DIS, from left to right: True value, MInet, Pointnet++.