Enhanced 2D Hand Pose Estimation for Gloved Medical Applications: A Preliminary Model

Abstract

(1) Background: As digital health technology evolves, the role of accurate medical-gloved hand tracking is becoming more important for the assessment and training of practitioners to reduce procedural errors in clinical settings. (2) Method: This study utilized computer vision for hand pose estimation to model skeletal hand movements during in situ aseptic drug compounding procedures. High-definition video cameras recorded hand movements while practitioners wore medical gloves of different colors. Hand poses were manually annotated, and machine learning models were developed and trained using the DeepLabCut interface via an 80/20 training/testing split. (3) Results: The developed model achieved an average root mean square error (RMSE) of 5.89 pixels across the training data set and 10.06 pixels across the test set. When excluding keypoints with a confidence value below 60%, the test set RMSE improved to 7.48 pixels, reflecting high accuracy in hand pose tracking. (4) Conclusions: The developed hand pose estimation model effectively tracks hand movements across both controlled and in situ drug compounding contexts, offering a first-of-its-kind medical glove hand tracking method. This model holds potential for enhancing clinical training and ensuring procedural safety, particularly in tasks requiring high precision such as drug compounding.

Keywords: aseptic technique; computer vision; drug compounding; hand tracking; machine learning; medical gloves; pose estimation.

Figure 1

A schematic of the camera placement across both data collections of the in situ data collection. For the initial training data collection, the left (L) and overhead (O) camera views were used, while only the L camera was used for the in situ testing data collection. The right (R) camera was used for the initial training data collection; however, no frames were coded for inclusion in the training data set from this perspective.

Figure 2

Comparisons between camera views: (A) camera position in the lower right corner of the LAFW, (B) camera position in the upper left of the LAFW, (C) overhead camera position within the LAFW.

Figure 3

Twenty-two keypoints identified and labeled on the right hand. Labels are mirrored on the contralateral hand and follow an identical naming convention.

Figure 4

Representation of average inference error in pixels. The center of each black cross indicates the ground-truth marker location, while the white circle area indicates average RMSE in pixels. Note, while the image is magnified for visibility, the circles are to scale relative to an RMSE based on the full 1920 × 1080 pixel image.

Figure 5

Error in manual labeling shown by overlaying two discrete images of the right hand from two different video frames, with the lighter and darker circles indicating independent labeling efforts of the same keypoint by the same human coder.

Figure 6

Video frame with annotations when the left hand occludes the right.