An Augmented Reality Periscope for Submarines with Extended Visual Classification

Abstract

Submarines are considered extremely strategic for any naval army due to their stealth capability. Periscopes are crucial sensors for these vessels, and emerging to the surface or periscope depth is required to identify visual contacts through this device. This maneuver has many procedures and usually has to be fast and agile to avoid exposure. This paper presents and implements a novel architecture for real submarine periscopes developed for future Brazilian naval fleet operations. Our system consists of a probe that is connected to the craft and carries a 360 camera. We project and take the images inside the vessel using traditional VR/XR devices. We also propose and implement an efficient computer vision-based MR technique to estimate and display detected vessels effectively and precisely. The vessel detection model is trained using synthetic images. So, we built and made available a dataset composed of 99,000 images. Finally, we also estimate distances of the classified elements, showing all the information in an AR-based interface. Although the probe is wired-connected, it allows for the vessel to stand in deep positions, reducing its exposure and introducing a new way for submarine maneuvers and operations. We validate our proposal through a user experience experiment using 19 experts in periscope operations.

Keywords: computer vision; deep learning; mixed reality; object detection; periscope; submarine; synthetic data; transfer learning.

Figure 1

Periscope exposure at periscope depth. Source: [2].

Figure 2

Overview of the proposed solution.

Figure 3

Our developed probe carrier mockup with a $${360}^{\circ }$$ camera mounted at the top of it.

Figure 4

Periscope point of view. Source: adapted from [17].

Figure 5

Classes of vessels considered while training the model using synthetic data; (a) Container Ship; (b) Ferry; (c) Frigate; (d) Passenger Ship; (e) Yard Ship.

Figure 6

Example of an augmented synthetic image used for training the classification module. This image includes Gaussian noise and blur.

Figure 7

Typical stadiometer split image.

Figure 8

Camera geometry.

Figure 9

Steps Flow Chart.

Figure 10

Labeled image. Blue means the box in which the object is contained.

Figure 11

Tool for helping vessels classification. The interface is in Portuguese.

Figure 12

Average loss during the first 12 K iterations.

Figure 13

Confusion Matrix.

Figure 14

Some results from the detection and classification model applied on synthetic images. The distance is estimated in meters, and the bow angle is given in degrees. (a) Container Ship at 2000 m and $${15}^{\circ }$$; (b) Ferry at 4000 m and $${160}^{\circ }$$; (c) Frigate at 2000 m and $${210}^{\circ }$$; (d) Passenger Ship at 4000 m and $${332}^{\circ }$$; (e) Yard Ship at 4000 m and $${332}^{\circ }$$.

Figure 15

Results of the detection and classification model applied on real images from the Internet. The classification probabilities are: (a) Container Ship, 0.98; (b) Ferry, 0.98; (c) Frigate, 0.92; (d) Passenger Ship, 0.98; and (e) Yard Ship, 0.99.

Figure 16

Results on two frames of a video sequence captured by our XR Periscope device. (a) Yard Ship at $$15.38$$ m; (b) Yard Ship at $$62.69$$ m.

Figure 17

User Interface of the XR Periscope.