Visual odometry based on structural matching of local invariant features using stereo camera sensor

Abstract

This paper describes a novel sensor system to estimate the motion of a stereo camera. Local invariant image features are matched between pairs of frames and linked into image trajectories at video rate, providing the so-called visual odometry, i.e., motion estimates from visual input alone. Our proposal conducts two matching sessions: the first one between sets of features associated to the images of the stereo pairs and the second one between sets of features associated to consecutive frames. With respect to previously proposed approaches, the main novelty of this proposal is that both matching algorithms are conducted by means of a fast matching algorithm which combines absolute and relative feature constraints. Finding the largest-valued set of mutually consistent matches is equivalent to finding the maximum-weighted clique on a graph. The stereo matching allows to represent the scene view as a graph which emerge from the features of the accepted clique. On the other hand, the frame-to-frame matching defines a graph whose vertices are features in 3D space. The efficiency of the approach is increased by minimizing the geometric and algebraic errors to estimate the final displacement of the stereo camera between consecutive acquired frames. The proposed approach has been tested for mobile robotics navigation purposes in real environments and using different features. Experimental results demonstrate the performance of the proposal, which could be applied in both industrial and service robot fields.

Keywords: combined constraint matching algorithm; maximum-weighted clique; robotic; stereo vision sensor; visual odometry sensor.

Figure 1.

Problem statement: given the pairs of stereo images taken at frames t − 1 and t, the robot motion is estimated from the natural landmarks {L}ⁱ. Two graphs emerge from the stereo and feature matching stages.

Figure 2.

Overview of the proposed visual odometry approach.

Figure 3.

(a) SIFT features found for the left and right images from the stereo image (F^l_t and F^r_t). The scale and orientation are indicated by the size and orientation of the vectors; (b) SURF features calculated using the stereo system in an outdoor environment. Scale are illustrated by the size of the circles (orientation is not shown in the figure).

Figure 4.

Vertices represent tentative matchings when considered individually. Arcs indicate compatible associations, and a clique is a set of mutually consistent associations (e.g., the clique {1, 5, 4} implies that associations f^1,l_t → f^1,r_t, f^2,l_t → f^2,r_t, f^3,l_t → f^3,r_t may coexist).

Figure 5.

Matched SIFT features between left and right images from the stereo pair shown in Figure 3. Red line represents matched points.

Figure 6.

Feature association results for two different displacements. After applying the maximum-weighted clique algorithm the number of pairwise matched features is 7 and 13 for the left and right images, respectively (3D coordinates of the landmarks are also included).

Figure 7.

A set of 320 × 240 images acquired by the camera has been used to evaluate the robustness and time processing of the matching algorithm. (a) a camera movement (translation and rotation); (b) a significant change in the scene; and (c) ambiguities due to similar objects in the scene.

Figure 8.

Performance of the matching algorithms used in the comparative study for various percentage of outliers. (a) True Positives against to different percentage of outliers; (b) Evolution of the precision against to different percentage of outliers; and (c) Time processing against the percentage of outliers. See the text for more details.

Figure 9.

Illustrative examples of the matching algorithm proposed in our visual odometry system for three different image tests used in the comparative study (results of the matching process for the images of the Figure 7(a–c), respectively). On the top, the initial matching which includes the 80% of outliers is shown. Below, results of the matching algorithm used in our approach have been drawn.

Figure 10.

Activmedia P2AT robot used in the experiments. (b–e) four different image pair acquired by the stereo camera across the robot motion in the first test. Stereo and feature matching are shown in the figure (red and green lines, respectively).

Figure 11.

Trajectories estimated by visual (Harris, SIFT and SURF) and wheel odometry (black, red, cyan and green line, respectively) for the first test. Blue lines define the trajectory estimated by the laser scan matching. Robot poses at the captured times shown in Figure 10 are labeled.

Figure 12.

(a–d) Four different image pairs acquired by the stereo camera across the robot motion in the second reported trial. Stereo and feature matching are shown in the figure (red and green line, respectively).

Figure 13.

Trajectories estimated by visual (Harris, SIFT and SURF) and wheel odometry (black, red, cyan and green lines, respectively) for the second reported test. Blue line defines the trajectory estimated by the laser scan matching. Blue dots represent the map obtained using the scan data acquired by the laser range finder. Robot poses at the captured times marked over Figure 12.

Figure 14.

(a) Trajectories estimated by visual and wheel odometry (black, red, cyan and green line, respectively) for the third test (outdoor scenario). Blue lines define the trajectory estimated by the laser scan matching; and (b), (c) two captures from the stereo camera and the results of the both matching processes.