Interactive video retrieval evaluation at a distance: comparing sixteen interactive video search systems in a remote setting at the 10th Video Browser Showdown

Abstract

The Video Browser Showdown addresses difficult video search challenges through an annual interactive evaluation campaign attracting research teams focusing on interactive video retrieval. The campaign aims to provide insights into the performance of participating interactive video retrieval systems, tested by selected search tasks on large video collections. For the first time in its ten year history, the Video Browser Showdown 2021 was organized in a fully remote setting and hosted a record number of sixteen scoring systems. In this paper, we describe the competition setting, tasks and results and give an overview of state-of-the-art methods used by the competing systems. By looking at query result logs provided by ten systems, we analyze differences in retrieval model performances and browsing times before a correct submission. Through advances in data gathering methodology and tools, we provide a comprehensive analysis of ad-hoc video search tasks, discuss results, task design and methodological challenges. We highlight that almost all top performing systems utilize some sort of joint embedding for text-image retrieval and enable specification of temporal context in queries for known-item search. Whereas a combination of these techniques drive the currently top performing systems, we identify several future challenges for interactive video search engines and the Video Browser Showdown competition itself.

Keywords: Content-based retrieval; Evaluations; Interactive video retrieval; Video browsing; Video content analysis.

Fig. 1

VBS2021 was organized as a fully virtual session

Fig. 2

Keyframes from a Visual KIS task, total duration of the video shown was 25 s. The task was solved by 13 out of 16 teams

Fig. 3

Time deltas of teams between first and last appearance of correct item in result logs before submission on shot level

Fig. 4

Relation between the rank of first occurrence of a shot in the result logs and time delta to correct submission. As expected, time delta increases with rank, with variance increasing as well

Fig. 5

Best rank of correct item appearing in result log. Teams are ordered by descending score on the x-axis. Different teams have used different thresholds to log results, for UX and performance reasons

Fig. 6

Green cells show the best achieved logged rank $r_s$ between 1 and 300 in time t of a correct scene frame in a task. The best rank $r_v$ of a correct video frame from the same result log is included, while $t_{\mathit{cs}}$ presents the time of the tool’s correct submission. Red values are for the best detected ranks of searched video frames if searched scene frames were not present in the logged result sets for a task. Red or orange cells show a browsing failure where the frame or video was retrieved but the team did not submit a correct result

Fig. 7

Share of AVS submissions judged as correct over time during an AVS task

Fig. 8

AVS submissions over time with a mean sliding window of size 25

Fig. 9

Cumulative unique correct video submissions over time during an AVS task

Fig. 10

Selected AVS metrics per task. Higher y-axis values indicate that for a task, teams found it easier to find results to submit

Fig. 11

Selected AVS metrics per task, looking at correct submissions. Higher y-axis values indicate that for a given task, it is easier to find results which judges deem correct

Fig. 12

Share of overall submissions per task over precision per team and task with the color indicating the evaluation metric score normalized over the best score of a task. For each task, all teams are represented as a dot

Fig. 13

Similar but differently judged AVS submissions, judged as incorrect (red border, on the left) vs. judged as correct (green border, on the right)