Asynchronous Semantic Background Subtraction

Abstract

The method of Semantic Background Subtraction (SBS), which combines semantic segmentation and background subtraction, has recently emerged for the task of segmenting moving objects in video sequences. While SBS has been shown to improve background subtraction, a major difficulty is that it combines two streams generated at different frame rates. This results in SBS operating at the slowest frame rate of the two streams, usually being the one of the semantic segmentation algorithm. We present a method, referred to as "Asynchronous Semantic Background Subtraction" (ASBS), able to combine a semantic segmentation algorithm with any background subtraction algorithm asynchronously. It achieves performances close to that of SBS while operating at the fastest possible frame rate, being the one of the background subtraction algorithm. Our method consists in analyzing the temporal evolution of pixel features to possibly replicate the decisions previously enforced by semantics when no semantic information is computed. We showcase ASBS with several background subtraction algorithms and also add a feedback mechanism that feeds the background model of the background subtraction algorithm to upgrade its updating strategy and, consequently, enhance the decision. Experiments show that we systematically improve the performance, even when the semantic stream has a much slower frame rate than the frame rate of the background subtraction algorithm. In addition, we establish that, with the help of ASBS, a real-time background subtraction algorithm, such as ViBe, stays real time and competes with some of the best non-real-time unsupervised background subtraction algorithms such as SuBSENSE.

Keywords: background subtraction; motion detection; scene labeling; semantic segmentation; video processing.

Figure 1

Timing diagram of a naive real-time implementation of the semantic background subtraction (SBS) method when the frame rate of semantics is too slow to handle all the frames in real time. From top to bottom, the time lines represent: the input frames $I_t$, the computation of semantics $S_t$ by the semantic segmentation algorithm (on GPU), the computation of intermediate segmentation masks $B_t$ by the background subtraction (BGS) algorithm (on CPU), and the computation of output segmentation masks $D_t$ by the SBS method (on CPU). Vertical lines indicate when an image is available and filled rectangular areas display when a GPU or CPU performs a task. Arrows show the inputs required by the different tasks. This diagram shows that even when the background subtraction algorithm is real time with respect to the input frame rate, it is the computation of semantics that dictates the output frame rate.

Figure 2

Schematic representation of our method named ASBS, extending SBS [30], capable to combine the two asynchronous streams of semantics and background subtraction masks to improve the performances of BGS algorithms. When semantics is available, Asynchronous Semantic Background Subtraction (ASBS) applies Rule 1, Rule 2, or selects the fallback, and it updates the color and rule maps. Otherwise, ASBS applies Rule A, Rule B, or it selects the fallback.

Figure 3

Timing diagram of ASBS in the case of a real-time BGS algorithm (${\Delta }_B<{\delta }_I$) satisfying the condition ${\Delta }_B+{\Delta }_D<{\delta }_I$. Note that the output stream is delayed by a constant ${\Delta }_S+{\Delta }_D$ time with respect to the input stream.

Figure 4

Overall $F_1$ scores obtained with SBS and ASBS for four state-of-the-art BGS algorithms and different sub-sampling factors. The performances of ASBS decrease much more slowly than those of SBS with the decrease of the semantic frame rate and, therefore, are much closer to those of the ideal case (SBS with all semantic maps computed, that is SBS 1:1), meaning that ASBS provides better decisions for frames without semantics. On average, ASBS with 1 frame of semantics out of 25 frames (ASBS $25:1$) performs as well as SBS, with copy of $B_t,$ with 1 frame of semantics out of 2 frames (SBS $2:1$).

Figure 5

Effects of SBS and ASBS on BGS algorithms in the mean ROC space of CDNet 2014 [12]. Each point represents the performance of a BGS algorithm and the end of the associated arrow indicates the performance after application of the methods for a temporal sub-sampling factor of $5:1$. We observe that SBS improves the performances, but only marginally, whereas ASBS moves the performances much closer to the oracle (upper left corner).

Figure 6

Per-category analysis. We display the relative improvements of the $F_1$ score of SBS, ASBS, and the second heuristic compared with the original algorithms, by considering only the frames without semantics (at a $5:1$ semantic frame rate).

Figure 7

Evolution of the optimal thresholds ${\tau }_A$ and ${\tau }_B$ of the ASBS method when the semantic frame rate is reduced. Note that the Manhattan distance associated to these thresholds is computed on 8-bit color values. The results are shown here for the PAWCS algorithm, and follow the same trend for the IUTIS-5, SuBSENSE, and WeSamBe BGS algorithms.

Figure 8

Our feedback mechanism, which impacts the decisions of any BGS algorithm whose model update is conservative, consists to replace the $\mathrm{BG}/\mathrm{FG}$ segmentation of the BGS algorithm by the final segmentation map improved by semantics (either by SBS or ASBS) to update the internal background model.

Figure 9

Comparison of the performances, computed with the mean $F_1$ score on the CDNet 2014, of SBS and ASBS when there is a feedback that uses $D_t$ to update the model of the BGS algorithm. The results are given with respect to a decreasing semantic frame rate. It can be seen that SBS and ASBS always improve the results of the original BGS algorithm and that a feedback is beneficial. Graphs in the right column show that the intrinsic quality of the BGS algorithms is improved, as their output $B_t$, prior to any combination with semantics, produces higher mean $F_1$ scores.

Figure 10

Illustration of the results of ASBS using ViBe as BGS algorithm. From left to right, we provide the original color image, the ground truth, the BGS as provided by the original ViBe algorithm, using our ASBS method without any feedback, and using ASBS and a feedback. Each line corresponds to a representative frame of a video in each category of CDNet2014.

Figure 11

Timing diagram of ASBS with a feedback mechanism in the case of a real-time BGS algorithm (${\Delta }_B<{\delta }_I$) satisfying the condition ${\Delta }_B+{\Delta }_D<{\delta }_I$ and the computation of semantics being not real-time (${\Delta }_S>{\delta }_I$). Note that the feedback time ${\Delta }_F$ is negligible.