Skip to main page content
U.S. flag

An official website of the United States government

Dot gov

The .gov means it’s official.
Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

Https

The site is secure.
The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Access keys NCBI Homepage MyNCBI Homepage Main Content Main Navigation
. 2019 Aug 14;15(2):88-94.
doi: 10.13004/kjnt.2019.15.e17. eCollection 2019 Oct.

Application of Deep Learning System into the Development of Communication Device for Quadriplegic Patient

Jung Hwan Lee  1 , Taewoo Kang  2 , Byung Kwan Choi  1 , In Ho Han  1 , Byung Chul Kim  1 , Jung Hoon Ro  3
Affiliations

Application of Deep Learning System into the Development of Communication Device for Quadriplegic Patient

Jung Hwan Lee et al. Korean J Neurotrauma. .

Abstract

Objective: In general, quadriplegic patients use their voices to call the caregiver. However, severe quadriplegic patients are in a state of tracheostomy, and cannot generate a voice. These patients require other communication tools to call caregivers. Recently, monitoring of eye status using artificial intelligence (AI) has been widely used in various fields. We made eye status monitoring system using deep learning, and developed a communication system for quadriplegic patients can call the caregiver.

Methods: The communication system consists of 3 programs. The first program was developed for automatic capturing of eye images from the face using a webcam. It continuously captured and stored 15 eye images per second. Secondly, the captured eye images were evaluated for open or closed status by deep learning, which is a type of AI. Google TensorFlow was used as a machine learning tool or library for convolutional neural network. A total of 18,000 images were used to train deep learning system. Finally, the program was developed to utter a sound when the left eye was closed for 3 seconds.

Results: The test accuracy of eye status was 98.7%. In practice, when the quadriplegic patient looked at the webcam and closed his left eye for 3 seconds, the sound for calling a caregiver was generated.

Conclusion: Our eye status detection software using AI is very accurate, and the calling system for the quadriplegic patient was satisfactory.

Keywords: Artificial intelligence; Caregiver; Eye; Quadriplegia; Unsupervised machine learning.

PubMed Disclaimer

Conflict of interest statement

Conflict of Interest: The authors have no financial conflicts of interest.

Figures

FIGURE 1
FIGURE 1. Both eyes are captured by the webcam.
FIGURE 2
FIGURE 2. Captured eye image. The size is 28×28 pixels, a total of 784 pixels.
FIGURE 3
FIGURE 3. Software coded using Google Tensorflow. The upper image is the algorithm related to sending and receiving eye images to the artificial intelligence server. The lower image is the algorithm related to generating a sound when the left eye was closed continuously for 3 seconds once the right eye was open.
FIGURE 4
FIGURE 4. Practical applications in a quadriplegic patient. (A) Actual setting for the quadriplegic patient. (B) Window of the software. Click ‘btn 1’ button to recognize both eyes. Click ‘log start’ button to recognize eye status, and call the caregiver. When the left eye was closed for 3 seconds, a sound was generated to call the caregiver. The ‘log end’ button was used to stop distinguishing eye status.
See this image and copyright information in PMC

Comment in

References

    1. Chang WD, Lim JH, Im CH. An unsupervised eye blink artifact detection method for real-time electroencephalogram processing. Physiol Meas. 2016;37:401–417. - PubMed
    1. Choi JS, Bang JW, Heo H, Park KR. Evaluation of fear using nonintrusive measurement of multimodal sensors. Sensors (Basel) 2015;15:17507–17533. - PMC - PubMed
    1. Dominguez Veiga JJ, O'Reilly M, Whelan D, Caulfield B, Ward TE. Feature-free activity classification of inertial sensor data with machine vision techniques: method, development, and evaluation. JMIR Mhealth Uhealth. 2017;5:e115. - PMC - PubMed
    1. Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017;542:115–118. - PMC - PubMed
    1. Feng R, Badgeley M, Mocco J, Oermann EK. Deep learning guided stroke management: a review of clinical applications. J Neurointerv Surg. 2018;10:358–362. - PubMed