Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2023 Feb 27;23(5):2623.
doi: 10.3390/s23052623.

3D Autonomous Surgeon's Hand Movement Assessment Using a Cascaded Fuzzy Supervisor in Multi-Thread Video Processing

Fatemeh Rashidi Fathabadi  1 , Janos L Grantner  1 , Saad A Shebrain  2 , Ikhlas Abdel-Qader  1
Affiliations

3D Autonomous Surgeon's Hand Movement Assessment Using a Cascaded Fuzzy Supervisor in Multi-Thread Video Processing

Fatemeh Rashidi Fathabadi et al. Sensors (Basel). .

Abstract

The purpose of the Fundamentals of Laparoscopic Surgery (FLS) training is to develop laparoscopic surgery skills by using simulation experiences. Several advanced training methods based on simulation have been created to enable training in a non-patient environment. Laparoscopic box trainers-cheap, portable devices-have been deployed for a while to offer training opportunities, competence evaluations, and performance reviews. However, the trainees must be under the supervision of medical experts who can evaluate their abilities, which is an expensive and time-consuming operation. Thus, a high level of surgical skill, determined by assessment, is necessary to prevent any intraoperative issues and malfunctions during a real laparoscopic procedure and during human intervention. To guarantee that the use of laparoscopic surgical training methods results in surgical skill improvement, it is necessary to measure and assess surgeons' skills during tests. We used our intelligent box-trainer system (IBTS) as a platform for skill training. The main aim of this study was to monitor the surgeon's hands' movement within a predefined field of interest. To evaluate the surgeons' hands' movement in 3D space, an autonomous evaluation system using two cameras and multi-thread video processing is proposed. This method works by detecting laparoscopic instruments and using a cascaded fuzzy logic assessment system. It is composed of two fuzzy logic systems executing in parallel. The first level assesses the left and right-hand movements simultaneously. Its outputs are cascaded by the final fuzzy logic assessment at the second level. This algorithm is completely autonomous and removes the need for any human monitoring or intervention. The experimental work included nine physicians (surgeons and residents) from the surgery and obstetrics/gynecology (OB/GYN) residency programs at WMU Homer Stryker MD School of Medicine (WMed) with different levels of laparoscopic skills and experience. They were recruited to participate in the peg-transfer task. The participants' performances were assessed, and the videos were recorded throughout the exercises. The results were delivered autonomously about 10 s after the experiments were concluded. In the future, we plan to increase the computing power of the IBTS to achieve real-time performance assessment.

Keywords: fuzzy logic-based decision support system; intelligent box-trainer system; laparoscopic surgical skill assessment; multi-class object detection.

PubMed Disclaimer

Conflict of interest statement

The authors declare no conflict of interest.

Figures

Figure 1
Figure 1
Pegboard with 12 pegs, and six ring-like objects (triangles) in the IBTS.
Figure 2
Figure 2
Top (a) and front (b) view segmentation for the purpose of three-dimensional assessment.
Figure 3
Figure 3
WMed surgical residents using the IBTS for laparoscopic training tasks.
Figure 4
Figure 4
(A). ResNet-50. (B) RPN network for generating regional proposals. In this proposed detector, the RPN connects the last conv feature map generated by FPN to a sliding window. It localizes any object, along with the RPN classifies scores and bounding boxes (Bbox) of the proposed regions. In part (C), the last conv feature map connects to a RoI pooling layer, leading to the proposed region. Finally, in (D) the classifier, there are two output layers of Fast R-CNN representing two vectors per proposed region: SoftMax probabilities and Bbox regression.
Figure 5
Figure 5
Block diagram of autonomous cascaded fuzzy supervisor assessment system in a multi-thread video processing experiment in the IBTS.
Figure 6
Figure 6
MFs of the input variable for the first level fuzzy logic evaluation system for both the right and left graspers.
Figure 7
Figure 7
Output variable MFs for surgeon’s performance assessment for the right and left-hand movements in the first level.
Figure 8
Figure 8
(a) MFs of input variables SRHPAr and SLHPAr, in the second level of the fuzzy logic evaluation system. (b) Membership functions of the output variable FPA in the second level.
Figure 9
Figure 9
The IBTS.
Figure 10
Figure 10
(a,b) Object detection and measurement metric results in top and front cameras, (c) SLHPA, (d) SRHPA, (e) final assessment FPA (good performance).
Figure 11
Figure 11
(a,b) Object detection and measurement metric results for top and front cameras, (c) SLHPA, (d) SRHPA, (e) final assessment FPA (not very good performance).
Figure 12
Figure 12
Object detection and measurement metric results by the top and front cameras, SLHPA, SRHPA, and final assessment FPA over a period of time.
Figure 12
Figure 12
Object detection and measurement metric results by the top and front cameras, SLHPA, SRHPA, and final assessment FPA over a period of time.
See this image and copyright information in PMC

References

    1. Haidegger T. Autonomy for surgical robots: Concepts and paradigms. IEEE Trans. Med. Robot. Bionics. 2019;1:65–76. doi: 10.1109/TMRB.2019.2913282. - DOI
    1. Peng J., Chen Q., Kang L., Jie H., Han Y. Autonomous Recognition of Multiple Surgical Instruments Tips Based on Arrow OBB-YOLO Network. IEEE Trans. Instrum. Meas. 2022;71:5007913. doi: 10.1109/TIM.2022.3162596. - DOI
    1. Chellali A., Mentis H., Miller A., Ahn W., Arikatla V.S., Sankaranarayanan G., De S., Schwaitzberg S.D., Cao C.G.L. Achieving interface and environment fidelity in the Virtual Basic Laparoscopic Surgical Trainer. Int. J. Hum. Comput. Stud. 2016;96:22–37. doi: 10.1016/j.ijhcs.2016.07.005. - DOI - PMC - PubMed
    1. Fathabadi F.R., Grantner J.L., Shebrain S.A., Abdel-Qader I. Surgical Skill Assessment System Using Fuzzy Logic in a Multi-Class Detection of Laparoscopic Box-Trainer Instruments; Proceedings of the 2021 IEEE International Conference on Systems, Man, and Cybernetics (SMC); Man, Melbourne, Australia. 17–20 October 2021; pp. 1248–1253.
    1. Oh D., Kwon C.H.D., Na B.G., Lee K.W., Cho W.T., Lee S.H., Choi J.Y., Choi G.S., Kim J.M., Joh J.-W. Surgical techniques for totally laparoscopic caudate lobectomy. J. Laparoendosc. Adv. Surg. Tech. 2016;26:689–692. doi: 10.1089/lap.2016.0161. - DOI - PubMed

LinkOut - more resources