Filipino sign language alphabet recognition using Persistent Homology Classification algorithm

Abstract

Increasing number of deaf or hard-of-hearing individuals is a crucial problem since communication among and within the deaf population proves to be a challenge. Despite sign languages developing in various countries, there is still lack of formal implementation of programs supporting its needs, especially for the Filipino sign language (FSL). Recently, studies on FSL recognition explored deep networks. Current findings are promising but drawbacks on using deep networks still prevail. This includes low transparency, interpretability, need for big data, and high computational requirements. Hence, this article explores topological data analysis (TDA), an emerging field of study that harnesses techniques from computational topology, for this task. Specifically, we evaluate a TDA-inspired classifier called Persistent Homology Classification algorithm (PHCA) to classify static alphabet signed using FSL and compare its result with classical classifiers. Experiment is implemented on balanced and imbalanced datasets with multiple trials, and hyperparameters are tuned for a comprehensive comparison. Results show that PHCA and support vector machine (SVM) performed better than the other classifiers, having mean Accuracy of 99.45% and 99.31%, respectively. Further analysis shows that PHCA's performance is not significantly different from SVM, indicating that PHCA performed at par with the best performing classifier. Misclassification analysis shows that PHCA struggles to classify signs with similar gestures, common to FSL recognition. Regardless, outcomes provide evidence on the robustness and stability of PHCA against perturbations to data and noise. It can be concluded that PHCA can serve as an alternative for FSL recognition, offering opportunities for further research.

Keywords: Classification algorithm; Filipino sign language; Persistent homology; Sign language recognition; Topological data analysis.

Figure 1. Illustration of a filtration of a point cloud into a nested sequence of simplicial complexes with K1⊆K2⊆…⊆K5.

Figure 2. Sphere and Torus point clouds and their corresponding persistent diagrams and barcodes obtained using persistent homology with Vietoris Rips filtration.

A persistent 2-dimensional hole (void) can be observed from the diagram and barcode of the sphere point cloud. Meanwhile, there is a persistent 1-dimensional hole (tunnel) that can be observed for the torus point cloud.

Figure 3. Framework of the classification scheme.

In (A), 63 features are extracted for each image in the dataset using the MediaPipe Hands pipeline. Then, data is split with stratification into 90–10 train/validation-test sets as shown in (B). Features are then scaled using Standard Scaler. In (C), hyperparameter tuning is employed to ensure best performance for each model. Using the hyperparameter combination that resulted with the best accuracy, the models are trained on the train/validation set and evaluated on the test set using five performance metrics. Comparison is implemented from this result.

Figure 4. Hand landmarks detected by the MediaPipe Hands pipeline.

Figure 5. Frequency of images converted and not converted into landmarks by the MediaPipe pipeline.

Class F and U are all converted into landmarks while class M and Q had the least number of converted images.

Figure 6. Distribution of performance metrics obtained by PHCA and the classical classifiers for the Balanced dataset.

Ten (10) trials are implemented, each with a different train-test split. Shown are the box-and-whisker plots of the resulting average precision (A), recall (B), F1-score (C), specificity (D), and overall accuracy (E). Shown also are the average metric values obtained by the classifiers across all 10 trials (F).

Figure 7. Distribution of performance metrics obtained by PHCA and the classical classifiers for the Imbalanced dataset.

Ten (10) trials are implemented, each with a different train-test split. Shown are the box-and-whisker plots of the resulting average precision (A), recall (B), F1-score (C), specificity (D), and overall accuracy (E). Shown also are the average metric values obtained by the classifiers across all 10 trials (F).

Figure 8. Confusion matrices associated with the predictions of PHCA for the Balanced and Imbalanced datasets.

The given matrices are the element-wise sum of the confusion matrix for each trial.

Figure 9. Sample images with landmarks that are often misclassified by PHCA.