A multimodal magnetoelastic artificial skin for underwater haptic sensing

Abstract

Future exploitation of marine resources in a sustainable and eco-friendly way requires autonomous underwater robotics with human-like perception. However, the development of such intelligent robots is now impeded by the lack of adequate underwater haptic sensing technology. Inspired by the populational coding strategy of the human tactile system, we harness the giant magnetoelasticity in soft polymer systems as an innovative platform technology to construct a multimodal underwater robotic skin for marine object recognition with intrinsic waterproofness and a simple configuration. The bioinspired magnetoelastic artificial skin enables multiplexed tactile modality in each single taxel and obtains an impressive classification rate of 95% in identifying seven types of marine creatures and marine litter. By introducing another degree of freedom in underwater haptic sensing, this work represents a milestone toward sustainable marine resource exploitation.

Fig. 1.. Overview of the BMAS-based underwater haptic sensing.

(A) The future exploitation of marine resources will encompass a range of activities including deep-sea biological sampling, mining, seafloor litter removal, and construction and maintenance of underwater infrastructure. To achieve these goals, autonomous underwater robots equipped with advanced haptic sensing capabilities will play a crucial role. (B) The untethered MC layer mimics the signal transmission functionality of the epidermis and dermis layers of human skin in response to applied tactile pressure. The two MI sensing layers resemble the tactile mechanoreceptors that are hierarchically distributed in the dermis. The sensation of amplitude, location, and type of the applied pressure on the artificial skin is realized by the cooperative response of multiple nearby coil pixels instead of a single one, which is imitating the human tactile sensing mechanism. (C) Illustration of the three layers of the magnetoelastic haptic sensing system. (D to F) Mechanism of the MC layer to distinguish the normal and shear force. Normal (E) and shear forces (F) yield a different magnetic field response, which can be picked by multiple sensory channels of the MI layer. k₁ and k₂ refers to effective elastic constant of the polymer matrix. (G) The customized circuit and algorithm design to mimic the encoding-decoding process of the human tactile system to realize multichannel magnetoelastic tactile perception and cognition.

Fig. 2.. Magnetic, mechanical, and electrical characterization of the magnetoelastic artificial skin.

(A) Stress-strain curve of the MC layer. (B) Displacement field of the unstructured MC layer when 20 kPa is applied to a pixel. Scale bar, 1 cm. (C) Displacement field of the groove-structured MC layer when 20 kPa is applied to a pixel. Scale bar, 1 cm. (D) Magnetic flux density versus applied normal stress at the center of a pixel. Data are means ± SD; N = 3. (E) Magnetic flux density variation versus applied shear stress at the center of a pixel and its nearby pixel. Data are means ± SD; N = 3. (F) Mapping of the Z-axis magnetic flux density at 0 kPa. Scale bar, 1 cm. (G) Mapping of the Z-axis magnetic flux density with 200 kPa applied to the white dashed box. Scale bar, 1 cm. (H) Mapping of the magnetic flux density using a magnetic flux display. Scale bar, 1 cm. (I) Mapping of the magnetic flux density with a pressing force of 5 N on the white dashed box using a magnetic flux display. Scale bar, 1 cm. (J) Illustration of normal stress on a single pixel and two sensory channels are excited. (K) Signal waveform in channel 2 and channel 6 under normal stress of 515.87 kPa. (L) Sensitivity of channel 2 and channel 6 under normal stress. Data are means ± SD; N = 5. (M) Illustration of shear stress on a single pixel and when three sensory channels are excited. (N) Signal waveform in channel 2, channel 6, and channel 1 under shear stress. (O) Sensitivity of channel 1 and channel 6 under shear stress. Data are means ± SD; N = 5. (P) Sensitivity of channel 2 and channel 6 under shear stress. Data are means ± SD; N = 5. A.U., arbitrary units.

Fig. 3.. Characterizing the perception capability of the magnetoelastic artificial skin.

(A) Electrical output of the magnetoelastic artificial skin under normal press at different pixels. (B) The reconstructed electrical output of 16 pixels in the artificial skin from the eight stacked orthogonal channels. (C) Reconstructed tactile mapping of the signal output intensity when writing “U,” “C,” “L,” and “A” and their corresponding voltage output profile using the eight stacked orthogonal channels. A.U., arbitrary units. (D) The electrical output of the magnetoelastic artificial skin under different tactile stimulation (press, shear right, shear down, shear left, shear, up, tapping, and vibration) at pixel 6.

Fig. 4.. The BMAS for underwater haptic sensing.

(A) Schematic of the BMAS integrated on an autonomous aquatic robot for marine litter perception and recognition. (B) Image of the prepared ocean creatures includes a green sea mollusk shell (C1), a turritellidae (C2), a scallop (C3), and a starfish (C4). (C) Image of the ocean debris includes a bottle cap (T1), a paper cup (T2), and a plastic bottle (T3). (D) Electrical output of eight channels for the recognition of a green sea mollusk shell. (E) Reconstructed mapping of the sea mollusk shell. A.U., arbitrary units. (F) Photograph of a robotic arm integrated with the BMAS through a customized clamp. Scale bar, 2 cm. (G) Representative photograph of objects randomly clamped by the BMAS-equipped robotic arm. Scale bars, 1 cm. (H) Architecture of the 1D convolutional neural network used for underwater object recognition. (I) Evaluation of the recognition loss function and accuracy of the 1D-CNN deep learning algorithm in 80 epochs. (J) Confusion matrix for the seven types of marine creatures and marine litter tactile pattern recognition with randomly selected orientations and contact positions. Actual class refers to the collected data, and predicted class refers to predictions using the 1D-CNN deep learning algorithm. The total classification accuracy reaches ~95%. (K) Probability distribution of identifying the green sea mollusk shell after Softmax step in the 1D-CNN. (L) t-SNE visualization results of the data from the seven marine objects showing the clustering feature. (M) Magnetic flux density and magnetoelastic response of the BMAS under different hydraulic pressure conditions from 0 to 25 mH₂O. Data are means ± SD; N = 3. (N) Variation of voltage amplitude of the BMAS immersed in 3.5 wt % NaCl solution over a 21-day period. C2, channel 2; C6, channel 6. Data are means ± 1.5 interquartile range; N = 4.