Hands Holding Clues for Object Recognition in Teachable Machines

Abstract

Camera manipulation confounds the use of object recognition applications by blind people. This is exacerbated when photos from this population are also used to train models, as with teachable machines, where out-of-frame or partially included objects against cluttered backgrounds degrade performance. Leveraging prior evidence on the ability of blind people to coordinate hand movements using proprioception, we propose a deep learning system that jointly models hand segmentation and object localization for object classification. We investigate the utility of hands as a natural interface for including and indicating the object of interest in the camera frame. We confirm the potential of this approach by analyzing existing datasets from people with visual impairments for object recognition. With a new publicly available egocentric dataset and an extensive error analysis, we provide insights into this approach in the context of teachable recognizers.

Keywords: blind; egocentric; hand; k-shot learning; object recognition.

Figure 1:

An illustration of our hand-guided object recognition approach on an example from our egocentric dataset. Given a photo of an object in proximity to a hand, it first identifies the hand and then estimates the object center, which is then cropped and passed to the recognition model.

Figure 2:

In our approach, a hand segmentation model (Step I) is fine-tuned to estimate the center of the object in proximity to the hand (Step II). A bounding box, placed in that center is used to isolate the object and crop the image, which is then passed to the object classification model (Step III).

Figure 3:

An (input, annotation, output) example for our hand segmentation (a) and object localization (b) models.

Figure 4:

Examples from each dataset. Glassense-Vision, VizWiz, and our benchmark examples are selected to include hands.

Figure 5:

Nineteen objects used in our data collection. Objects in the same category are displayed in proximity.

Figure 6:

Positive and negative outputs of our object localization model on the Glassense-Vision and VizWiz datasets.

Figure 7:

Our hand-guided object recognition method (CO) tends to improve recognition accuracy on average for S and B compared to the original HO and O methods.

Figure 8:

Accuracy gain of our method (CO) over HO and O is more pertinent in cluttered backgrounds (wild).

Figure 9:

On average CO outperforms HO and O consistently across training sample sizes k = 1, 5, 20.

Figure 10:

Presence of hands (HO and CO) seems to have a different efect for generic vs. teachable models.

Figure 11:

Positive and negative results on TEgO, with outof-frame hands for some of the negative examples.

Figure 12:

Confusion matrix for the CO models showing that misclassifcation occurs within objects of similar shape. Cans and bottles are indicated as “-c” and “-b”, respectively.