Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2017 Mar;6(1):1-29.
doi: 10.1007/s13735-017-0121-3. Epub 2017 Feb 22.

Instance Search Retrospective with Focus on TRECVID

George Awad  1 , Wessel Kraaij  2 , Paul Over  3 , Shin'ichi Satoh  4
Affiliations

Instance Search Retrospective with Focus on TRECVID

George Awad et al. Int J Multimed Inf Retr. 2017 Mar.

Abstract

This paper presents an overview of the Video Instance Search benchmark which was run over a period of 6 years (2010-2015) as part of the TREC Video Retrieval (TRECVID) workshop series. The main contributions of the paper include i) an examination of the evolving design of the evaluation framework and its components (system tasks, data, measures); ii) an analysis of the influence of topic characteristics (such as rigid/non rigid, planar/non-planar, stationary/mobile on performance; iii) a high-level overview of results and best-performing approaches. The Instance Search (INS) benchmark worked with a variety of large collections of data including Sound & Vision, Flickr, BBC (British Broadcasting Corporation) Rushes for the first 3 pilot years and with the small world of the BBC Eastenders series for the last 3 years.

Keywords: TRECVID; evaluation; instance search; multimedia.

PubMed Disclaimer

Figures

Fig. 1
Fig. 1
Standard processing flow of instance search
Fig. 2
Fig. 2
Example frames from the different datasets used
Fig. 3
Fig. 3
Examples of pilot evaluation topics (Objects, Persons, Locations)
Fig. 4
Fig. 4
2013: Examples of BBC Eastenders topics (Objects, Persons)
Fig. 5
Fig. 5
2014: Examples of BBC Eastenders topics (Objects, Persons, Location)
Fig. 6
Fig. 6
2015: Examples of BBC Eastenders topics (Objects, Persons, Location)
Fig. 7
Fig. 7
2011: MAP vs. elapsed time
Fig. 8
Fig. 8
2012: MAP vs. elapsed time
Fig. 9
Fig. 9
2013: MAP vs. elapsed time
Fig. 10
Fig. 10
2014: MAP vs. elapsed time for fastest runs (in seconds)
Fig. 11
Fig. 11
2015: MAP vs. elapsed time
Fig. 12
Fig. 12
2011: True positives per topic vs. maximum average precision
Fig. 13
Fig. 13
2012: True positives per topic vs. maximum average precision
Fig. 14
Fig. 14
2013: True positives per topic vs. maximum average precision
Fig. 15
Fig. 15
2014: True positives per topic vs. maximum average precision
Fig. 16
Fig. 16
2015: True positives per topic vs. maximum average precision
Fig. 17
Fig. 17
2010: Average precision for automatic runs by topic/type
Fig. 18
Fig. 18
2011: Average precision for automatic runs by topic/type
Fig. 19
Fig. 19
2011: Average precision for interactive runs by topic/type
Fig. 20
Fig. 20
2011: AP by topic for top runs
Fig. 21
Fig. 21
2012: Average precision for automatic runs by topic/type
Fig. 22
Fig. 22
2012: Average precision for interactive runs by topic/type
Fig. 23
Fig. 23
2012: AP by topic for top runs
Fig. 24
Fig. 24
2013: Boxplot of automatic runs - average precision by topic
Fig. 25
Fig. 25
2013: Boxplot of interactive runs - average precision by topic
Fig. 26
Fig. 26
2014: Boxplot of average precision by topic for automatic runs
Fig. 27
Fig. 27
2014: Boxplot of average precision by topic for interactive runs
Fig. 28
Fig. 28
2014: Effect of number of topic example images used
Fig. 29
Fig. 29
2015: Boxplot of average precision by topic for automatic runs
Fig. 30
Fig. 30
2015: Boxplot of average precision by topic for interactive runs
Fig. 31
Fig. 31
2015: Automatic results by example Sets (image-only vs video+image)
Fig. 32
Fig. 32
2013–2015: Mobile vs. Stationary
Fig. 33
Fig. 33
2013: Box plots of score distributions per topic type (A,B,C,D)-(M,S) pair
Fig. 34
Fig. 34
2014: Box plots of score distributions per topic type (A,B,C,D)-(M,S) pair
Fig. 35
Fig. 35
2015: Box plots of score distributions per topic type (A,B,C,D)-(M,S) pair
Fig. 36
Fig. 36
Examples of queries with lowest median AP scores
See this image and copyright information in PMC

References

    1. Arandjelovic O, Zisserman A. Automatic face recognition for film character retrieval in feature-length films. Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on; 2005. pp. 860–867. - DOI
    1. Arandjelovic R, Zisserman A. In: Bowden R, Collomosse JP, Mikolajczyk K, editors. Multiple queries for large scale specific object retrieval; British Machine Vision Conference, BMVC 2012, Surrey, UK; September 3–7, 2012; BMVA Press; 2012. pp. 1–11. URL http://dx.doi.org/10.5244/C.26.92. - DOI - DOI
    1. Arandjelovic R, Zisserman A. Three things everyone should know to improve object retrieval; 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA; June 16–21, 2012; IEEE Computer Society; 2012. pp. 2911–2918. URL http://dx.doi.org/10.1109/CVPR.2012.6248018. - DOI - DOI
    1. Babenko A, Lempitsky VS. Aggregating local deep features for image retrieval; 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile; December 7–13, 2015; IEEE Computer Society; 2015. pp. 1269–1277. URL http://dx.doi.org/10.1109/ICCV.2015.150. - DOI - DOI
    1. Babenko A, Slesarev A, Chigorin A, Lempitsky VS. In: Fleet DJ, Pajdla T, Schiele B, Tuytelaars T, editors. Neural codes for image retrieval; Computer Vision - ECCV 2014 - 13th European Conference, Zurich, Switzerland, September 6–12, 2014, Proceedings, Part I, Lecture Notes in Computer Science; Springer; 2014. pp. 584–599. URL http://dx.doi.org/10.1007/978-3-319-10590-1_38. - DOI - DOI

LinkOut - more resources