Prediction of Shooting Events in Soccer Videos Using Complete Bipartite Graphs and Players' Spatial-Temporal Relations

Abstract

In soccer, quantitatively evaluating the performance of players and teams is essential to improve tactical coaching and players' decision-making abilities. To achieve this, some methods use predicted probabilities of shoot event occurrences to quantify player performances, but conventional shoot prediction models have not performed well and have failed to consider the reliability of the event probability. This paper proposes a novel method that effectively utilizes players' spatio-temporal relations and prediction uncertainty to predict shoot event occurrences with greater accuracy and robustness. Specifically, we represent players' relations as a complete bipartite graph, which effectively incorporates soccer domain knowledge, and capture latent features by applying a graph convolutional recurrent neural network (GCRNN) to the constructed graph. Our model utilizes a Bayesian neural network to predict the probability of shoot event occurrence, considering spatio-temporal relations between players and prediction uncertainty. In our experiments, we confirmed that the proposed method outperformed several other methods in terms of prediction performance, and we found that considering players' distances significantly affects the prediction accuracy.

Keywords: Bayesian neural network; graph convolutional recurrent neural network; shoot event prediction; soccer video; spatio-temporal information.

Figure 1

This section provides an overview of our pre-processing approach. Firstly, we use a full-frame image (a) as input and obtain a Stroke Width Transform (SWT) image (b) and a field mask (c) through SWT and a pix2pix-cGan segmentation generator network, respectively. This leads to an edge image (d) of the field. Next, we capture the corresponding camera parameters by searching for the nearest neighbor field edge image (e) based on HOG features. In another step, we classify the detected players into their respective teams based on HSV color. Finally, using both pieces of information, we obtain the players’ positions on the field as well as their teams.

Figure 2

Perspective projection camera model. First, the coordinates of the field edge $(x_w,y_w,z_w)$ in the world coordinate system are transformed to their coordinates $(x_c,y_c,z_c)$ in the camera coordinate system, which originates from the camera center. Then, we obtain the projected coordinates $(x_n,y_n)$ in the normalized image plane of size $u\times v$.

Figure 3

Standard soccer field dimensions. These are expressed in meters for each measurement.

Figure 4

Overview of the proposed method. Our method starts by extracting visual features of each detected player and full-frame at each time step form a given soccer video input. A complete bipartite graph is constructed with these visual features as node features. We then use GCRNN to calculate latent features. Next, our method predicts the probability of shoot event occurrence based on BNN, and we obtain the resulting predictions along with their corresponding prediction uncertainties. The overall process is illustrated in Figure.

Figure 5

The GCRNN architecture comprises stacked GCN layers that initially learn the spatial relations. By inputting the RNN’s hidden state ${\mathit{h}}_{t-1}$ into the second GCN layer, the method can calculate the latent feature ${\mathit{Z}}_t$, considering both temporal and spatial relations. This process in repeated recursively to capture latent features at each time step, as illustrated in Figure.

Figure 6

The qualitative prediction results of the proposed method for shoot event occurrence at a threshold of $s=0.5$. It displays an example of a positive sample with successful predictions. The proposed method predicts the probability of shoot event occurrence at each time step (indicated by the blue curve) and estimates both aleatoric uncertainty (shown in the orange region) and epistemic uncertainty (shown in the yellow region) for each time step.

Figure 7

This figure illustrates an instance of accurate predictions for the positive sample. The details depicted are identical to those in Figure 6.

Figure 8

This figure presents an example of precise predictions for the negative sample. The specifics depicted are identical to those in Figure 6.

Figure 9

This figure demonstrates an instance of inaccurate predictions for the negative sample. The particulars shown are identical to those in Figure 6.

Figure 10

This figure displays an instance of erroneous predictions for the positive sample. The specifics depicted are identical to those in Figure 6.