Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2021 Sep-Oct:2021:9773-9779.
doi: 10.1109/iros51168.2021.9636640. Epub 2021 Dec 16.

NYU-VPR: Long-Term Visual Place Recognition Benchmark with View Direction and Data Anonymization Influences

Diwei Sheng  1 , Yuxiang Chai  1 , Xinru Li  1 , Chen Feng  1 , Jianzhe Lin  1 , Claudio Silva  1 , John-Ross Rizzo  1
Affiliations

NYU-VPR: Long-Term Visual Place Recognition Benchmark with View Direction and Data Anonymization Influences

Diwei Sheng et al. Rep U S. 2021 Sep-Oct.

Abstract

Visual place recognition (VPR) is critical in not only localization and mapping for autonomous driving vehicles, but also assistive navigation for the visually impaired population. To enable a long-term VPR system on a large scale, several challenges need to be addressed. First, different applications could require different image view directions, such as front views for self-driving cars while side views for the low vision people. Second, VPR in metropolitan scenes can often cause privacy concerns due to the imaging of pedestrian and vehicle identity information, calling for the need for data anonymization before VPR queries and database construction. Both factors could lead to VPR performance variations that are not well understood yet. To study their influences, we present the NYU-VPR dataset that contains more than 200,000 images over a 2km×2km area near the New York University campus, taken within the whole year of 2016. We present benchmark results on several popular VPR algorithms showing that side views are significantly more challenging for current VPR methods while the influence of data anonymization is almost negligible, together with our hypothetical explanations and in-depth analysis.

PubMed Disclaimer

Figures

Fig. 1.
Fig. 1.
Top-1 VPR retrieval accuracy of 4 baseline algorithms under different view directions and data anonymization.
Fig. 2.
Fig. 2.
Dataset visualization of NYU-VPR w.r.t. the image capturing time.
Fig. 3.
Fig. 3.
Raw images vs. Anonymized images.
Fig. 4.
Fig. 4.
Dataset visualization of NYU-VPR with respect to the image capturing location. The locations of (a)-(c) are highlighted in (d).
Fig. 5.
Fig. 5.
Further explanation and illustration of our dataset.
Fig. 6.
Fig. 6.
Main results of the benchmark.
Fig. 7.
Fig. 7.
We visualize VPR results only for VLAD+SuperPoint, VLAD+SURF, and DBOW, because the result of NetVLAD is very similar to VLAD+SuperPoint. We randomly picked one location (more locations results in the supplementary). The first row is for the anonymized front-view query. The second row is for the corresponding raw front-view query. The third row is for the side-view query taken at the same location as the front-view query. The last row is for the corresponding raw side-view query. The red cross means the location of the retrieval image is not in the distance threshold (10 meters). We show the top 3 retrievals for each method.
Fig. 8.
Fig. 8.
VPR success rate vs. anonymized rate. The same legend as Fig. 6.
Fig. 9.
Fig. 9.
VPR success rate vs. image blurriness.
Fig. 10.
Fig. 10.
VPR success rate vs. query image month.
See this image and copyright information in PMC

References

    1. Mirowski P, Banki-Horvath A, Anderson K, Teplyashin D, Hermann KM, Malinowski M, Grimes MK, Simonyan K, Kavukcuoglu K, Zisserman A et al., “The streetlearn environment and dataset,” arXiv preprint arXiv:1903.01292, 2019. 2
    1. Zamir AR and Shah M, “Image geo-localization based on multiplenearest neighbor feature matching usinggeneralized graphs,” IEEE Trans. Pattern Anal. Mach. Intell, vol. 36, no. 8, pp. 1546–1558, 2014. 2 - PubMed
    1. Sünderhauf N, Neubert P, and Protzel P, “Are we there yet? challenging seqslam on a 3000 km journey across all four seasons,” in Proc. IEEE Int’l Conf. Robotics and Automation (ICRA), 2013, p. 2013. 2, 3
    1. “The VPRiCE Challenge 2015 – Visual Place Recognition in Changing Environments - Public - Confluence.” [Online]. Available: https://roboticvision.atlassian.net/wiki/spaces/PUB/pages/14188617/The+V...
    1. Torii A, Arandjelovic R, Sivic J, Okutomi M, and Pajdla T, “24/7 place recognition by view synthesis,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2015, pp. 1808–1817. 2, 3 - PubMed

LinkOut - more resources