A Spatial-Spectral Classification Method Based on Deep Learning for Controlling Pelagic Fish Landings in Chile

Abstract

Fishing has provided mankind with a protein-rich source of food and labor, allowing for the development of an important industry, which has led to the overexploitation of most targeted fish species. The sustainable management of these natural resources requires effective control of fish landings and, therefore, an accurate calculation of fishing quotas. This work proposes a deep learning-based spatial-spectral method to classify five pelagic species of interest for the Chilean fishing industry, including the targeted Engraulis ringens, Merluccius gayi, and Strangomera bentincki and non-targeted Normanichthtys crockeri and Stromateus stellatus fish species. This proof-of-concept method is composed of two channels of a convolutional neural network (CNN) architecture that processes the Red-Green-Blue (RGB) images and the visible and near-infrared (VIS-NIR) reflectance spectra of each species. The classification results of the CNN model achieved over 94% in all performance metrics, outperforming other state-of-the-art techniques. These results support the potential use of the proposed method to automatically monitor fish landings and, therefore, ensure compliance with the established fishing quotas.

Keywords: VIS-NIR; deep learning; fish; hyperspectral imaging; image processing; machine learning.

Figure 1

Setup of the hyperspectral imaging system for reflectance measurements. Note that the moving platform is indicated by the red arrows.

Figure 2

Sample separation from an image of a set of fish.

Figure 3

Whole-body average reflectance curves of the five species studied.

Figure 4

Extraction of average spectral signatures from a single fish hypercube. Note that the random points of the RoIs are indicated by numbers in the fish image.

Figure 5

Generation of RGB images from the fish hypercubes.

Figure 6

Architecture of CNN for the classification of pelagic species. The CNN has two input channels that independently extract features from the spatial and spectral channels (input data). The dimension flow of the CNN network is also provided in bold text.

Figure 7

Example of image data augmentation for Engraulis ringens. (a) Original: $1070\times 260\times 3$ pixels. (b) Rescaling: $256\times 256\times 3$ pixels. (c) Final image: $256\times 256\times 3$ pixels.

Figure 8

Example of spectral data augmentation for Engraulis ringens. (a,b) Spectral signatures (average reflectance) using two different sets of RoIs, respectively.

Figure 9

Comparative results of the training and test error for the best model using each feature type (input).

Figure 10

Comparative results for each problem class.