Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2020 May 5:3:32.
doi: 10.3389/frai.2020.00032. eCollection 2020.

Evaluating Multimedia and Language Tasks

Ian Soboroff  1 , George Awad  1   2 , Asad Butt  1   3 , Keith Curtis  1
Affiliations

Evaluating Multimedia and Language Tasks

Ian Soboroff et al. Front Artif Intell. .

Abstract

Evaluating information access tasks, including textual and multimedia search, question answering, and understanding has been the core mission of NIST's Retrieval Group since 1989. The TRECVID Evaluations of Multimedia Access began in 2001 with a goal of driving content-based search technology for multimedia just as its progenitor, the Text Retrieval Conference (TREC) did for text and web.

Keywords: annotation; evaluation; information retrieval (IR); metrics; multimedia.

PubMed Disclaimer

Figures

Figure 1
Figure 1
TRECVID teams, authors, and peer-reviewed academic publications by year.
Figure 2
Figure 2
Comparison of video captions between the MSVD dataset and our TRECVID VTT dataset. The top row shows screenshots of two videos from the MSVD dataset, along with five captions. The bottom row shows screenshots of comparable videos from the TRECVID dataset and their corresponding captions.
See this image and copyright information in PMC

References

    1. Abualsaud M., Ghelani N., Zhang H., Smucker M. D., Cormack G. V., Grossman M. R. (2018). “A system for efficient high-recall retrieval,” in The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR '18 (New York, NY: Association for Computing Machinery; ), 1317–1320.
    1. Allan J., Aslam J., Belkin N., Buckley C., Callan J., Croft B., et al. (2003). Challenges in information retrieval and language modeling. SIGIR Forum 37 10.1145/945546.945549 - DOI
    1. Anderson P., Fernando B., Johnson M., Gould S. (2016). “Spice: Semantic propositional image caption evaluation,” in European Conference on Computer Vision (Amsterdam: Springer; ), 382–398.
    1. Antol S., Agrawal A., Lu J., Mitchell M., Batra D., Lawrence Zitnick C., et al. (2015). “VQA: Visual question answering,” in Proceedings of the IEEE International Conference on Computer Vision (Santiago: ), 2425–2433.
    1. Aslam J. A., Pavlu V., Yilmaz E. (2006). “A statistical method for system evaluation using incomplete judgments,” in Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR'06 (New York, NY: Association for Computing Machinery; ), 541–548.

LinkOut - more resources