Identifying daily activities of patient work for type 2 diabetes and co-morbidities: a deep learning and wearable camera approach

Abstract

Objective: People are increasingly encouraged to self-manage their chronic conditions; however, many struggle to practise it effectively. Most studies that investigate patient work (ie, tasks involved in self-management and contexts influencing such tasks) rely on self-reports, which are subject to recall and other biases. Few studies use wearable cameras and deep learning to capture and classify patient work activities automatically.

Materials and methods: We propose a deep learning approach to classify activities of patient work collected from wearable cameras, thereby studying self-management routines more effectively. Twenty-six people with type 2 diabetes and comorbidities wore a wearable camera for a day, generating more than 400 h of video across 12 daily activities. To classify these video images, a weighted ensemble network that combines Linear Discriminant Analysis, Deep Convolutional Neural Networks, and Object Detection algorithms is developed. Performance of our model is assessed using Top-1 and Top-5 metrics, compared against manual classification conducted by 2 independent researchers.

Results: Across 12 daily activities, our model achieved on average the best Top-1 and Top-5 scores of 81.9 and 86.8, respectively. Our model also outperformed other non-ensemble techniques in terms of Top-1 and Top-5 scores for most activity classes, demonstrating the superiority of leveraging weighted ensemble techniques.

Conclusions: Deep learning can be used to automatically classify daily activities of patient work collected from wearable cameras with high levels of accuracy. Using wearable cameras and a deep learning approach can offer an alternative approach to investigate patient work, one not subjected to biases commonly associated with self-report methods.

Keywords: deep learning; patient work; self-management; wearable camera.

Figure 1.

An overview of our approach. It consists of 4 steps: Dataset Preparation, Sleeping Activity Classification, Non-Sleeping Activities Classification, and Weighted Ensemble Network.

Figure 2.

Flowchart and detailed illustration for each step involved in our approach.

Figure 3.

Confusion matrix of our weighted ensemble network for the 12 classes with rows as the predicted labels and columns as the actual labels.

Figure 4.

Example classification results of our weighted ensemble network for patients 03, 12, 18, 26. The class in bold corresponds to the ground truth labels.

Figure 5.

Complete illustrations of daily activities for patients 03, 12, 28, 26.